You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vinothchandar (via GitHub)" <gi...@apache.org> on 2026/08/05 13:46:02 UTC
[PR] feat(core): spark-parity writer: all write operations, table types, query types, markers, archival, locks, MDT indexes, parity harness [hudi-rs]
vinothchandar opened a new pull request, #666:
URL: https://github.com/apache/hudi-rs/pull/666
## Description
Bring the native write path to storage parity with the Apache Hudi Spark
writer (Hudi 1.1.x, table version 9 default; tv8 retained) and prove it with
a Spark-in-the-loop parity harness.
Writer:
- Two-phase locking per write (LockProvider + InProcessLockProvider):
request instant under lock, complete + bookkeeping under lock.
- Skew-adjusted monotonic instant times, formatted in the table's declared
timeline timezone (LOCAL default, matching Spark) so interleaved writers
produce ordered timelines.
- Timeline-server-based marker files and eager rollback of failed writes
(rollback plan/metadata, orphan MDT cleanup).
- LSM timeline archival (manifest_N + _version_ + level-0 parquet) with
reader-side archival-floor and fencing unions.
- Small-file packing (UpsertPartitioner semantics) and group-aware COW
rewrites; parallel write tasks via hoodie.write.task.parallelism.
- MDT files/column_stats/partition_stats/record_index maintained per commit;
MetadataPayloadMerger mirrors HoodieMetadataPayload combine semantics and
is resolved from the MDT merge-strategy id.
- Deletes that empty a file group write an empty base file version; commit
metadata stores the data schema without meta fields; batches differing
only in nullability are aligned.
- MDT records decode with the writer schema of their source container, so
mixed-writer tables (rs + Spark) read correctly.
- Incremental queries follow Hudi 1.x completion-time semantics with exact
instant-set filtering, shared by table.read and manual reader paths.
Parity harness (make parity, HUDI_SPARK_PARITY=1, scripts/parity/):
six scenarios against a local Spark 3.5 + Hudi bundle — rs<->Spark reads and
writes, Spark compaction/clustering/clean on rs tables followed by rs writes,
interleaved writers, a 36-commit mixed workload with archival verified via
snapshot/time-travel/incremental in both engines, and a CDC-enabled table.
Diff size justified: this lands the writer feature set plus its test
matrix (six new integration suites and the parity harness).
## How are the changes test-covered
- [ ] N/A
- [ ] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4393444799
##########
README.md:
##########
@@ -322,6 +322,107 @@ std::vector<std::string> log_file_paths{};
ArrowArrayStream* stream_ptr = reader->read_file_slice_from_paths("relative/path.parquet", log_file_paths);
```
+## Writing Tables (Rust)
Review Comment:
For README.md, could we restructure the doc with read and write sections? Right now `Usage Examples` and `Writing Tables (Rust)` are at the same level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4395061489
##########
crates/core/src/metadata/table/column_stats.rs:
##########
@@ -0,0 +1,638 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Extract column-range metadata from Parquet footers for MDT writes.
+
+use std::collections::HashMap;
+
+use arrow_schema::{DataType, Schema, TimeUnit};
+use bytes::Bytes;
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+use crate::Result;
+use crate::error::CoreError;
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::{ColumnStatValue, ColumnStatsMetadata};
+
+/// Per-column range extracted from a written Parquet base file.
+#[derive(Debug, Clone)]
+pub struct ColumnRangeStats {
+ pub column_name: String,
+ pub min_value: Option<ColumnStatValue>,
+ pub max_value: Option<ColumnStatValue>,
+ /// False when the parquet footer gave truncated/absent statistics, so the
+ /// bounds are UNKNOWN rather than absent. Readers treat a published record
+ /// with null min/max as "all nulls" and prune the file, so a record with
+ /// an unknown range must not be published at all (the file then falls into
+ /// the reader's not-indexed rescue set).
+ pub range_known: bool,
+ pub value_count: i64,
+ pub null_count: i64,
+ pub total_size: i64,
+ pub total_uncompressed_size: i64,
+}
+
+/// Read file-level column ranges from in-memory Parquet bytes (Java `ParquetUtils`).
+///
+/// Covers exactly [`columns_to_index`]: the three always-indexed meta fields plus
+/// the first 32 supported data fields. Nested (multi-part path) columns and types
+/// Spark does not index (bytes/fixed/decimal) are omitted rather than mis-encoded.
+pub fn column_ranges_from_parquet_bytes(bytes: &[u8]) -> Result<Vec<ColumnRangeStats>> {
+ let builder = ParquetRecordBatchReaderBuilder::try_new(Bytes::copy_from_slice(bytes))
+ .map_err(|e| CoreError::Write(format!("Failed to open Parquet for column stats: {e}")))?;
+ let schema = builder.schema().clone();
+ let metadata = builder.metadata().clone();
+ Ok(column_ranges_from_parquet_metadata(
+ metadata.as_ref(),
+ schema.as_ref(),
+ ))
+}
+
+fn column_ranges_from_parquet_metadata(
+ metadata: &parquet::file::metadata::ParquetMetaData,
+ schema: &Schema,
+) -> Vec<ColumnRangeStats> {
+ let type_by_name: HashMap<&str, &DataType> = schema
+ .fields()
+ .iter()
+ .map(|f| (f.name().as_str(), f.data_type()))
+ .collect();
+ let indexed_columns: std::collections::HashSet<String> =
+ columns_to_index(schema).into_iter().collect();
+
+ #[derive(Default)]
+ struct Acc {
+ min: Option<ColumnStatValue>,
+ max: Option<ColumnStatValue>,
+ value_count: i64,
+ null_count: i64,
+ total_size: i64,
+ total_uncompressed_size: i64,
+ data_type: Option<DataType>,
+ /// A chunk carried no statistics at all, so the accumulated range does
+ /// not cover the whole column and must not be published as a bound.
+ /// Truncated bounds do NOT set this: parquet truncates min downward
+ /// and max upward, so they stay valid (just not tight) bounds, and
+ /// Java publishes them as-is (ParquetUtils.readColumnStatsFromMetadata).
+ range_incomplete: bool,
+ }
+
+ let mut by_col: HashMap<String, Acc> = HashMap::new();
+ for row_group in metadata.row_groups() {
+ for col_chunk in row_group.columns() {
+ let path = col_chunk.column_descr().path();
+ if path.parts().len() != 1 {
+ continue;
+ }
+ let Some(col_name) = path.parts().first().map(|s| s.as_str()) else {
+ continue;
+ };
+ if !indexed_columns.contains(col_name) {
+ continue;
+ }
+ let Some(&data_type) = type_by_name.get(col_name) else {
+ continue;
+ };
+
+ let acc = by_col.entry(col_name.to_string()).or_default();
+ acc.data_type = Some(data_type.clone());
+ acc.value_count += col_chunk.num_values();
+ acc.total_size += col_chunk.compressed_size();
+ acc.total_uncompressed_size += col_chunk.uncompressed_size();
+
+ if let Some(stats) = col_chunk.statistics() {
+ // Null count only when actually recorded; never fabricate
+ // "all null" for chunks with stats disabled.
+ acc.null_count += stats.null_count_opt().unwrap_or(0) as i64;
+ match parquet_min_max_to_stat_values(stats, data_type) {
+ // Truncated bounds are still bounds; publish them (the
+ // record is never marked tight).
+ Some((min, max)) => {
+ acc.min = merge_min(acc.min.take(), Some(min));
+ acc.max = merge_max(acc.max.take(), Some(max));
+ }
+ // Parquet emits stubbed stats for an all-null chunk: no
+ // bounds exist, and none are needed — it cannot widen the
+ // column's range (Java equates nulls to the value count).
+ None if stats
+ .null_count_opt()
+ .is_some_and(|nulls| nulls as i64 == col_chunk.num_values()) => {}
+ None => acc.range_incomplete = true,
+ }
+ } else {
+ // Stats disabled for the chunk: range and null count unknown.
+ acc.range_incomplete = true;
+ }
+ }
+ }
+
+ let mut out: Vec<ColumnRangeStats> = by_col
+ .into_iter()
+ .map(|(column_name, acc)| ColumnRangeStats {
+ column_name,
+ range_known: !acc.range_incomplete,
+ min_value: (!acc.range_incomplete).then_some(acc.min).flatten(),
+ max_value: (!acc.range_incomplete).then_some(acc.max).flatten(),
+ value_count: acc.value_count,
+ null_count: acc.null_count,
+ total_size: acc.total_size,
+ total_uncompressed_size: acc.total_uncompressed_size,
+ })
+ .collect();
+ out.sort_by(|a, b| a.column_name.cmp(&b.column_name));
+ out
+}
+
+/// Build MDT column_stats payloads for a written (or deleted) file.
+pub fn column_stats_for_file(
+ file_name: &str,
+ ranges: &[ColumnRangeStats],
+ is_deleted: bool,
+) -> Vec<ColumnStatsMetadata> {
+ ranges
+ .iter()
+ .filter(|r| is_deleted || r.range_known || r.null_count == r.value_count)
+ .map(|r| ColumnStatsMetadata {
+ file_name: file_name.to_string(),
+ column_name: r.column_name.clone(),
+ min_value: if is_deleted {
+ None
+ } else {
+ r.min_value.clone()
+ },
+ max_value: if is_deleted {
+ None
+ } else {
+ r.max_value.clone()
+ },
+ value_count: if is_deleted { 0 } else { r.value_count },
+ null_count: if is_deleted { 0 } else { r.null_count },
+ total_size: if is_deleted { 0 } else { r.total_size },
+ total_uncompressed_size: if is_deleted {
+ 0
+ } else {
+ r.total_uncompressed_size
+ },
+ is_deleted,
+ is_tight_bound: false,
+ decoded_value_type_ordinal: None,
+ })
+ .collect()
+}
+
+/// Tombstone column_stats for a deleted file when footer ranges are unavailable.
+pub fn column_stats_tombstones(
+ file_name: &str,
+ column_names: &[String],
+) -> Vec<ColumnStatsMetadata> {
+ column_names
+ .iter()
+ .map(|column_name| ColumnStatsMetadata {
+ file_name: file_name.to_string(),
+ column_name: column_name.clone(),
+ min_value: None,
+ max_value: None,
+ value_count: 0,
+ null_count: 0,
+ total_size: 0,
+ total_uncompressed_size: 0,
+ is_deleted: true,
+ is_tight_bound: false,
+ decoded_value_type_ordinal: None,
+ })
+ .collect()
+}
+
+/// Aggregate file-level ranges into partition-level stats (same column name).
+pub fn aggregate_partition_stats(
+ partition_path: &str,
+ ranges_by_file: &[Vec<ColumnRangeStats>],
+ is_tight_bound: bool,
+) -> Vec<ColumnStatsMetadata> {
+ #[derive(Default)]
+ struct Acc {
+ min: Option<ColumnStatValue>,
+ max: Option<ColumnStatValue>,
+ value_count: i64,
+ null_count: i64,
+ total_size: i64,
+ total_uncompressed_size: i64,
+ range_incomplete: bool,
+ }
+ let mut by_col: HashMap<String, Acc> = HashMap::new();
+ for ranges in ranges_by_file {
+ for r in ranges {
+ let acc = by_col.entry(r.column_name.clone()).or_default();
+ // One file with unknown bounds makes the partition aggregate a
+ // non-tight (advisory) bound: publishing it tight would let
+ // pruning skip a partition whose unknown file may match.
+ if !r.range_known && r.null_count != r.value_count {
+ acc.range_incomplete = true;
+ }
+ acc.min = merge_min(acc.min.take(), r.min_value.clone());
+ acc.max = merge_max(acc.max.take(), r.max_value.clone());
+ acc.value_count += r.value_count;
+ acc.null_count += r.null_count;
+ acc.total_size += r.total_size;
+ acc.total_uncompressed_size += r.total_uncompressed_size;
+ }
+ }
+ // Java partition_stats stores the partition path in fileName.
+ let file_name = if partition_path.is_empty() {
+ ".".to_string()
+ } else {
+ partition_path.to_string()
+ };
+ let mut out: Vec<ColumnStatsMetadata> = by_col
+ .into_iter()
+ // A partition record with null bounds reads as "all nulls" and Spark's
+ // partition pruning has no not-indexed rescue set, so it would prune
+ // the whole partition. Publish nothing instead — unless the column
+ // really is all nulls, where null bounds are the correct encoding.
+ .filter(|(_, acc)| {
+ acc.min.is_some() || acc.max.is_some() || acc.null_count == acc.value_count
+ })
+ .map(|(column_name, acc)| ColumnStatsMetadata {
+ file_name: file_name.clone(),
+ column_name,
+ min_value: acc.min,
+ max_value: acc.max,
+ value_count: acc.value_count,
+ null_count: acc.null_count,
+ total_size: acc.total_size,
+ total_uncompressed_size: acc.total_uncompressed_size,
+ is_deleted: false,
+ is_tight_bound: is_tight_bound && !acc.range_incomplete,
+ decoded_value_type_ordinal: None,
+ })
+ .collect();
+ out.sort_by(|a, b| a.column_name.cmp(&b.column_name));
+ out
+}
+
+/// Java `HoodieMetadataConfig.COLUMN_STATS_INDEX_MAX_COLUMNS` default: only the
+/// first 32 supported data fields (schema order) are indexed.
+pub const MAX_COLUMNS_TO_INDEX: usize = 32;
+
+/// Columns eligible for MDT indexing from a table/data schema.
+///
+/// Java `HoodieTableMetadataUtil.getColumnsToIndex`: the three always-indexed
+/// meta fields (when present, i.e. `populateMetaFields`) plus the first
+/// [`MAX_COLUMNS_TO_INDEX`] supported non-meta fields in schema order.
+pub fn columns_to_index(schema: &Schema) -> Vec<String> {
+ // Java META_COLS_TO_ALWAYS_INDEX_SCHEMA_MAP is a TreeMap: alphabetical order.
+ let mut cols: Vec<String> = [
+ MetaField::CommitTime,
+ MetaField::PartitionPath,
+ MetaField::RecordKey,
+ ]
+ .iter()
+ .map(|m| m.as_ref().to_string())
+ .filter(|name| schema.fields().iter().any(|f| f.name() == name))
+ .collect();
+ cols.extend(
+ schema
+ .fields()
+ .iter()
+ .filter(|f| !MetaField::is_meta_field(f.name()) && is_indexable_type(f.data_type()))
+ .take(MAX_COLUMNS_TO_INDEX)
+ .map(|f| f.name().clone()),
+ );
+ cols
+}
+
+/// Java `isColumnTypeSupported` for the SPARK record type: nested types, enums,
+/// raw bytes/fixed — and hence decimals, stored as bytes/fixed — are not indexed.
+fn is_indexable_type(data_type: &DataType) -> bool {
+ matches!(
+ data_type,
+ DataType::Boolean
+ | DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::UInt8
+ | DataType::UInt16
+ | DataType::UInt32
+ | DataType::UInt64
+ | DataType::Float32
+ | DataType::Float64
+ | DataType::Utf8
+ | DataType::LargeUtf8
+ | DataType::Date32
+ | DataType::Time64(TimeUnit::Microsecond)
+ | DataType::Timestamp(_, _)
+ )
+}
+
+fn parquet_min_max_to_stat_values(
+ stats: &ParquetStatistics,
+ data_type: &DataType,
+) -> Option<(ColumnStatValue, ColumnStatValue)> {
+ match (stats, data_type) {
+ (ParquetStatistics::Boolean(s), DataType::Boolean) => Some((
+ ColumnStatValue::Boolean(*s.min_opt()?),
+ ColumnStatValue::Boolean(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Int32(s), DataType::Date32) => Some((
+ ColumnStatValue::Date(*s.min_opt()?),
+ ColumnStatValue::Date(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Int32(s), _) => Some((
+ ColumnStatValue::Int(*s.min_opt()?),
+ ColumnStatValue::Int(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Int64(s), DataType::Timestamp(unit, tz)) => {
+ let min = to_micros(*s.min_opt()?, *unit)?;
+ let max = to_micros(*s.max_opt()?, *unit)?;
+ // tz-less arrow timestamps are Java LOCAL_TIMESTAMP_* semantics.
+ if tz.is_some() {
+ Some((
+ ColumnStatValue::TimestampMicros(min),
+ ColumnStatValue::TimestampMicros(max),
+ ))
+ } else {
+ Some((
+ ColumnStatValue::LocalTimestampMicros(min),
+ ColumnStatValue::LocalTimestampMicros(max),
+ ))
+ }
+ }
+ (ParquetStatistics::Int64(s), DataType::Time64(TimeUnit::Microsecond)) => Some((
+ ColumnStatValue::TimeMicros(*s.min_opt()?),
+ ColumnStatValue::TimeMicros(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Int64(s), _) => Some((
+ ColumnStatValue::Long(*s.min_opt()?),
+ ColumnStatValue::Long(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Float(s), _) => Some((
+ ColumnStatValue::Float(*s.min_opt()?),
+ ColumnStatValue::Float(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Double(s), _) => Some((
+ ColumnStatValue::Double(*s.min_opt()?),
+ ColumnStatValue::Double(*s.max_opt()?),
+ )),
+ (ParquetStatistics::ByteArray(s), DataType::Utf8 | DataType::LargeUtf8) => {
+ let min = String::from_utf8_lossy(s.min_opt()?.data()).into_owned();
+ let max = String::from_utf8_lossy(s.max_opt()?.data()).into_owned();
+ Some((ColumnStatValue::String(min), ColumnStatValue::String(max)))
+ }
+ (ParquetStatistics::ByteArray(s), _) => Some((
+ ColumnStatValue::Bytes(s.min_opt()?.data().to_vec()),
+ ColumnStatValue::Bytes(s.max_opt()?.data().to_vec()),
+ )),
+ (ParquetStatistics::FixedLenByteArray(s), _) => Some((
+ ColumnStatValue::Bytes(s.min_opt()?.data().to_vec()),
+ ColumnStatValue::Bytes(s.max_opt()?.data().to_vec()),
+ )),
+ _ => None,
+ }
+}
+
+fn to_micros(value: i64, unit: TimeUnit) -> Option<i64> {
+ match unit {
+ TimeUnit::Second => value.checked_mul(1_000_000),
+ TimeUnit::Millisecond => value.checked_mul(1_000),
+ TimeUnit::Microsecond => Some(value),
+ TimeUnit::Nanosecond => Some(value / 1_000),
+ }
+}
+
+fn merge_min(a: Option<ColumnStatValue>, b: Option<ColumnStatValue>) -> Option<ColumnStatValue> {
Review Comment:
The derived `PartialOrd` on `ColumnStatValue` orders by variant before value, so when surviving MDT stats and new footer stats decode to different variants, min and max each pick a side regardless of value (appending ts=100 then ts=10 on tv8 gave a tight-bound record with min=100, max=10). I pushed [c93661fd](https://github.com/apache/hudi-rs/pull/666/commits/c93661fde41166c8d448f00616444c07fabf072b), which converts both sides to the column's type before merging and errors instead of emitting a tight bound it cannot compute.
##########
crates/core/src/metadata/table/mod.rs:
##########
@@ -455,6 +469,7 @@ impl Table {
let data_pending = self.timeline.all_pending_instant_times().await?;
let mut valid: HashSet<String> = self.valid_from_completed_data_instants();
+ valid.extend(self.archived_data_instants().await?);
Review Comment:
This source reads the full LSM history on every MDT-backed read (including Java-written v8+ tables), and a history read error fails the snapshot read; it's also redundant, since archived data instants are never pending and `valid_from_mdt_delta_commits` already admits their MDT deltacommits. I pushed [3b465d94](https://github.com/apache/hudi-rs/pull/666/commits/3b465d947a76b6ec279ceba260721259e5c81317), which drops it and admits MDT bases below the earliest active instant by the same rule `is_committed` uses, with a test that reads correctly with a corrupted history.
##########
crates/core/src/write/append.rs:
##########
@@ -0,0 +1,1227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::collections::HashMap;
+use std::io::Cursor;
+use std::str::FromStr;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicI64, Ordering};
+
+use arrow::array::{ArrayRef, StringArray, UInt32Array};
+use arrow::record_batch::RecordBatch;
+use arrow_select::take::take;
+use chrono::Utc;
+use parquet::arrow::ArrowWriter;
+
+use crate::Result;
+use crate::config::table::HudiTableConfig::{
+ PopulatesMetaFields, RecordMergeStrategy, TableType, TableVersion, TimelineLayoutVersion,
+ TimelinePath,
+};
+use crate::config::table::TableTypeValue;
+use crate::error::CoreError;
+use crate::index::is_record_index_enabled;
+use crate::merge::RecordMergeStrategyValue;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::RecordIndexEntry;
+use crate::table::Table;
+use crate::timeline::instant::{Action, Instant};
+use crate::write::keygen::{
+ hoodie_keys_for_batch, hoodie_keys_for_batch_with_offset, relative_data_path,
+};
+use crate::write::metadata::{
+ is_column_stats_enabled, update_column_stats_partitions, update_files_partition,
+ update_record_index,
+};
+
+/// Result of an append write.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct AppendResult {
+ pub instant: String,
+ pub commit_relative_path: String,
+ pub base_file_path: String,
+ pub num_rows: usize,
+}
+
+static LAST_EPOCH_MILLIS: AtomicI64 = AtomicI64::new(0);
+
+/// Append one or more record batches as a single INSERT commit.
+pub async fn append_batches(table: &mut Table, batches: &[RecordBatch]) -> Result<AppendResult> {
+ append_batches_inner(table, batches, false).await
+}
+
+/// Append one or more record batches, requiring append-only merge mode.
+pub async fn append_batches_only(
+ table: &mut Table,
+ batches: &[RecordBatch],
+) -> Result<AppendResult> {
+ append_batches_inner(table, batches, true).await
+}
+
+async fn append_batches_inner(
+ table: &mut Table,
+ batches: &[RecordBatch],
+ require_append_only: bool,
+) -> Result<AppendResult> {
+ table.reload_timeline_for_write().await?;
+ if !table.is_mor() {
+ if require_append_only {
+ ensure_append_only_strategy(table)?;
+ } else {
+ ensure_insert_allowed(table)?;
+ }
+ }
+ ensure_supported_table_type(table)?;
+
+ if batches.is_empty() {
+ return Err(CoreError::Write(
+ "append requires at least one RecordBatch".to_string(),
+ ));
+ }
+ let schema = batches[0].schema();
+ for batch in batches.iter().skip(1) {
+ if batch.schema().as_ref() != schema.as_ref() {
+ return Err(CoreError::Write(
+ "All RecordBatches in append must share the same schema".to_string(),
+ ));
+ }
+ }
+ ensure_append_schema_matches_table(table, schema.as_ref()).await?;
+ // Must be expressible in commit metadata (Avro JSON) before fencing.
+ arrow_schema_to_avro_json(&strip_meta_fields_from_schema(schema.as_ref()))?;
+ let num_rows: usize = batches.iter().map(|b| b.num_rows()).sum();
+ if num_rows == 0 {
+ return Err(CoreError::Write(
+ "append requires at least one row".to_string(),
+ ));
+ }
+
+ crate::write::keygen::validate_keygen_inputs(&table.hudi_configs, batches)?;
+
+ let layout_two = is_layout_two(table);
+ let action = if table.is_mor() {
+ Action::DeltaCommit
+ } else {
+ Action::Commit
+ };
+ let timeline_dir = timeline_dir(table);
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ // Critical section 1: mint the instant and fence it, then work unlocked.
+ let cs1 = lock.lock().await?;
+ let request_instant = generate_instant_time().await;
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir,
+ &request_instant,
+ action.clone(),
+ Vec::new(),
+ crate::write::inflight_commit_metadata_bytes("INSERT", layout_two)?,
+ )
+ .await?;
+ drop(cs1);
+
+ // Group rows across batches by partition path and write one base file per partition.
+ let mut partition_rows: HashMap<String, Vec<(usize, usize)>> = HashMap::new();
+ for (batch_idx, batch) in batches.iter().enumerate() {
+ let keys = hoodie_keys_for_batch(&table.hudi_configs, batch, Some(&request_instant))?;
+ for (row_idx, key) in keys.iter().enumerate() {
+ partition_rows
+ .entry(key.partition_path.clone())
+ .or_default()
+ .push((batch_idx, row_idx));
+ }
+ }
+
+ let mut write_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ let mut files_mdt = Vec::new();
+ let mut rli_entries = Vec::new();
+ let mut stats_updates = Vec::new();
+ let mut written_paths: Vec<String> = Vec::new();
+ let mut primary_base_path = String::new();
+ let max_records = max_records_per_file(table);
+ // Monotonic auto-key row id across partitions and size-split chunks in this commit.
+ let mut auto_key_row_offset = 0usize;
+ let mut auto_key_partition_id = 0u32;
+
+ // Stable partition iteration so partition_id assignment is deterministic.
+ let mut partition_rows_sorted: Vec<_> = partition_rows.into_iter().collect();
+ partition_rows_sorted.sort_by(|a, b| a.0.cmp(&b.0));
+
+ // Plan every output file name up front and write markers before any data
+ // file is put — rollback of a crashed write is driven by these markers.
+ let mut file_plan: HashMap<String, std::collections::VecDeque<(String, String)>> =
+ HashMap::new();
+ let mut planned_markers = Vec::new();
+ for (partition_path, row_refs) in &partition_rows_sorted {
+ let chunks = row_refs.len().div_ceil(max_records).max(1);
+ let entries = file_plan.entry(partition_path.clone()).or_default();
+ for _ in 0..chunks {
+ let file_id = crate::write::new_file_id();
+ let file_name = format!("{file_id}_0-0-0_{request_instant}.parquet");
+ planned_markers.push(crate::write::markers::Marker::create(
+ partition_path,
+ &file_name,
+ ));
+ entries.push_back((file_id, file_name));
+ }
+ }
+ crate::write::markers::write_markers(storage.as_ref(), &request_instant, &planned_markers)
+ .await?;
+
+ // Phase 1 (sequential): slice + prepare every chunk, keeping table borrows
+ // out of the parallel phase.
+ struct ChunkPlan {
+ partition_path: String,
+ file_id: String,
+ file_name: String,
+ base_file_path: String,
+ row_count: i64,
+ chunk_keys: Vec<crate::index::HoodieKey>,
+ prepared: Vec<RecordBatch>,
+ }
+ let mut chunk_plans: Vec<ChunkPlan> = Vec::new();
+ for (partition_path, row_refs) in partition_rows_sorted {
+ // BTreeMap: deterministic row order in the written file.
+ let batch_indices: std::collections::BTreeMap<usize, Vec<u32>> = {
+ let mut map: std::collections::BTreeMap<usize, Vec<u32>> =
+ std::collections::BTreeMap::new();
+ for (batch_idx, row_idx) in &row_refs {
+ map.entry(*batch_idx).or_default().push(*row_idx as u32);
+ }
+ map
+ };
+ let mut partition_batches = Vec::new();
+ for (batch_idx, rows) in batch_indices {
+ partition_batches.push(take_rows(&batches[batch_idx], &rows)?);
+ }
+ let partition_batch = match partition_batches.pop() {
+ Some(only) if partition_batches.is_empty() => only,
+ Some(last) => {
+ partition_batches.push(last);
+ arrow::compute::concat_batches(&schema, &partition_batches)?
+ }
+ None => continue,
+ };
+ crate::write::ensure_partition_metadata(
+ storage.as_ref(),
+ &partition_path,
+ &request_instant,
+ )
+ .await?;
+
+ // Split oversized partitions into multiple base files (Java max.file.size).
+ let total_rows = partition_batch.num_rows();
+ let mut offset = 0usize;
+ while offset < total_rows {
+ let end = (offset + max_records).min(total_rows);
+ let indices: Vec<u32> = (offset as u32..end as u32).collect();
+ let chunk = take_rows(&partition_batch, &indices)?;
+ let chunk_row_offset = auto_key_row_offset;
+ let chunk_keys = hoodie_keys_for_batch_with_offset(
+ &table.hudi_configs,
+ &chunk,
+ Some(&request_instant),
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ auto_key_row_offset += chunk.num_rows();
+ let (file_id, file_name) = file_plan
+ .get_mut(&partition_path)
+ .and_then(std::collections::VecDeque::pop_front)
+ .ok_or_else(|| {
+ CoreError::Write(format!(
+ "file plan exhausted for partition '{partition_path}'"
+ ))
+ })?;
+ let base_file_path = relative_data_path(&partition_path, &file_name);
+ if primary_base_path.is_empty() {
+ primary_base_path = base_file_path.clone();
+ }
+ let prepared = prepare_batches_for_write_with_offset(
+ &table.hudi_configs,
+ &[chunk],
+ &request_instant,
+ &file_name,
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ chunk_plans.push(ChunkPlan {
+ partition_path: partition_path.clone(),
+ file_id,
+ file_name,
+ base_file_path,
+ row_count: indices.len() as i64,
+ chunk_keys,
+ prepared,
+ });
+ offset = end;
+ }
+ auto_key_partition_id = auto_key_partition_id.saturating_add(1);
+ }
+
+ // Phase 2 (parallel): encode + put each file on the write task pool.
+ let props = parquet_writer_props(table);
+ let collect_ranges = is_column_stats_enabled(table);
+ let tasks = chunk_plans
+ .iter()
+ .map(|plan| {
+ crate::write::parquet_file_task(
+ storage.clone(),
+ props.clone(),
+ plan.prepared.clone(),
+ plan.base_file_path.clone(),
+ collect_ranges,
+ )
+ })
+ .collect();
+ let results =
+ crate::write::run_write_tasks(tasks, crate::write::write_task_parallelism(table)).await;
+
+ // Phase 3 (sequential): assemble outputs in plan order; clean up on error.
+ let mut first_error = None;
+ for (plan, result) in chunk_plans.iter().zip(results) {
+ match result {
+ Ok(output) => {
+ written_paths.push(plan.base_file_path.clone());
+ if let Some(ranges) = output.ranges {
+ stats_updates.push(crate::write::metadata::StatsFileUpdate {
+ partition_path: plan.partition_path.clone(),
+ file_name: plan.file_name.clone(),
+ is_deleted: false,
+ ranges,
+ });
+ }
+ write_stats
+ .entry(plan.partition_path.clone())
+ .or_default()
+ .push(HoodieWriteStat {
+ file_id: Some(plan.file_id.clone()),
+ path: Some(plan.base_file_path.clone()),
+ // Basename only — FileGroup builder joins partition_path + name.
+ base_file: Some(plan.file_name.clone()),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(plan.row_count),
+ num_inserts: Some(plan.row_count),
+ total_write_bytes: Some(output.size),
+ file_size_in_bytes: Some(output.size),
+ partition_path: Some(plan.partition_path.clone()),
+ ..Default::default()
+ });
+ files_mdt.push((
+ plan.partition_path.clone(),
+ plan.file_name.clone(),
+ output.size,
+ ));
+ for key in &plan.chunk_keys {
+ rli_entries.push(RecordIndexEntry {
+ record_key: key.record_key.clone(),
+ partition_path: plan.partition_path.clone(),
+ file_id: plan.file_id.clone(),
+ instant_time_millis: crate::write::metadata::instant_to_epoch_millis_in(
+ &request_instant,
+ &crate::write::metadata::timeline_timezone(&table.hudi_configs),
+ ),
+ is_deleted: false,
+ });
+ }
+ }
+ Err(error) => {
+ if first_error.is_none() {
+ first_error = Some(error);
+ }
+ }
+ }
+ }
+ if let Some(error) = first_error {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+
+ let commit_metadata = HoodieCommitMetadata {
+ version: Some(1),
+ operation_type: Some("INSERT".to_string()),
+ partition_to_write_stats: Some(write_stats),
+ compacted: Some(false),
+ // Java HoodieCommitMetadata.extraMetadata is non-null; null → NPE in Spark.
+ extra_metadata: Some(HashMap::from([(
+ "schema".to_string(),
+ arrow_schema_to_avro_json(&schema)?,
+ )])),
+ };
+ let commit_bytes = if layout_two {
+ commit_metadata.to_avro_bytes()?
+ } else {
+ commit_metadata.to_json_bytes()?
+ };
+
+ // Crash-consistency order: data files → MDT files → MDT deltacommit →
+ // completed data commit. Finalizing the data timeline last means readers
+ // fence MDT contents by completed data instants. MDT log-file writes are
+ // "work" (unlocked); only completion runs in critical section 2.
+ let mut mdt_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ if table.is_metadata_table_enabled() {
+ let mdt_result = async {
+ mdt_stats.insert(
+ "files".to_string(),
+ update_files_partition(storage.as_ref(), &request_instant, &files_mdt).await?,
+ );
+ if is_record_index_enabled(table) {
+ let stats =
+ update_record_index(storage.as_ref(), &request_instant, &rli_entries).await?;
+ if !stats.is_empty() {
+ mdt_stats.insert("record_index".to_string(), stats);
+ }
+ }
+ if is_column_stats_enabled(table) && !stats_updates.is_empty() {
+ let stats = update_column_stats_partitions(
+ table,
+ &request_instant,
+ &stats_updates,
+ schema.as_ref(),
+ &[],
+ )
+ .await?;
+ mdt_stats.extend(stats);
+ }
+ Ok::<(), crate::error::CoreError>(())
+ }
+ .await;
+ if let Err(error) = mdt_result {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ }
+
+ // Critical section 2: complete the action (MDT deltacommit + data commit
+ // with a completion time minted under the lock), then bookkeeping.
+ let cs2 = lock.lock().await?;
+ if table.is_metadata_table_enabled()
+ && let Err(error) = crate::write::metadata::write_metadata_commit(
+ storage.as_ref(),
+ &request_instant,
+ mdt_stats,
+ )
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ let completion = if layout_two {
+ Some(generate_instant_time().await)
+ } else {
+ None
+ };
+ let instant = Instant::new_completed(request_instant.clone(), action, completion)?;
+ let commit_relative_path = instant.relative_path_with_base(&timeline_dir)?;
+ if let Err(error) = storage
Review Comment:
This deletes the data files when the completed-commit PUT errors, but on object stores that outcome is ambiguous: a timed-out or retried conditional PUT can fail even though the commit landed, leaving a completed commit that points at deleted files. I pushed [8180631b](https://github.com/apache/hudi-rs/pull/666/commits/8180631b2515728f921da35223a5b1d37a20bdeb), which keeps the files once the completion PUT has been attempted (here and in the COW/MOR completion paths) and leaves cleanup to the next write's eager rollback, with a fault-injection test.
##########
crates/core/src/write/append.rs:
##########
@@ -0,0 +1,1227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::collections::HashMap;
+use std::io::Cursor;
+use std::str::FromStr;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicI64, Ordering};
+
+use arrow::array::{ArrayRef, StringArray, UInt32Array};
+use arrow::record_batch::RecordBatch;
+use arrow_select::take::take;
+use chrono::Utc;
+use parquet::arrow::ArrowWriter;
+
+use crate::Result;
+use crate::config::table::HudiTableConfig::{
+ PopulatesMetaFields, RecordMergeStrategy, TableType, TableVersion, TimelineLayoutVersion,
+ TimelinePath,
+};
+use crate::config::table::TableTypeValue;
+use crate::error::CoreError;
+use crate::index::is_record_index_enabled;
+use crate::merge::RecordMergeStrategyValue;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::RecordIndexEntry;
+use crate::table::Table;
+use crate::timeline::instant::{Action, Instant};
+use crate::write::keygen::{
+ hoodie_keys_for_batch, hoodie_keys_for_batch_with_offset, relative_data_path,
+};
+use crate::write::metadata::{
+ is_column_stats_enabled, update_column_stats_partitions, update_files_partition,
+ update_record_index,
+};
+
+/// Result of an append write.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct AppendResult {
+ pub instant: String,
+ pub commit_relative_path: String,
+ pub base_file_path: String,
+ pub num_rows: usize,
+}
+
+static LAST_EPOCH_MILLIS: AtomicI64 = AtomicI64::new(0);
+
+/// Append one or more record batches as a single INSERT commit.
+pub async fn append_batches(table: &mut Table, batches: &[RecordBatch]) -> Result<AppendResult> {
+ append_batches_inner(table, batches, false).await
+}
+
+/// Append one or more record batches, requiring append-only merge mode.
+pub async fn append_batches_only(
+ table: &mut Table,
+ batches: &[RecordBatch],
+) -> Result<AppendResult> {
+ append_batches_inner(table, batches, true).await
+}
+
+async fn append_batches_inner(
+ table: &mut Table,
+ batches: &[RecordBatch],
+ require_append_only: bool,
+) -> Result<AppendResult> {
+ table.reload_timeline_for_write().await?;
+ if !table.is_mor() {
+ if require_append_only {
+ ensure_append_only_strategy(table)?;
+ } else {
+ ensure_insert_allowed(table)?;
+ }
+ }
+ ensure_supported_table_type(table)?;
+
+ if batches.is_empty() {
+ return Err(CoreError::Write(
+ "append requires at least one RecordBatch".to_string(),
+ ));
+ }
+ let schema = batches[0].schema();
+ for batch in batches.iter().skip(1) {
+ if batch.schema().as_ref() != schema.as_ref() {
+ return Err(CoreError::Write(
+ "All RecordBatches in append must share the same schema".to_string(),
+ ));
+ }
+ }
+ ensure_append_schema_matches_table(table, schema.as_ref()).await?;
+ // Must be expressible in commit metadata (Avro JSON) before fencing.
+ arrow_schema_to_avro_json(&strip_meta_fields_from_schema(schema.as_ref()))?;
+ let num_rows: usize = batches.iter().map(|b| b.num_rows()).sum();
+ if num_rows == 0 {
+ return Err(CoreError::Write(
+ "append requires at least one row".to_string(),
+ ));
+ }
+
+ crate::write::keygen::validate_keygen_inputs(&table.hudi_configs, batches)?;
+
+ let layout_two = is_layout_two(table);
+ let action = if table.is_mor() {
+ Action::DeltaCommit
+ } else {
+ Action::Commit
+ };
+ let timeline_dir = timeline_dir(table);
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ // Critical section 1: mint the instant and fence it, then work unlocked.
+ let cs1 = lock.lock().await?;
+ let request_instant = generate_instant_time().await;
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir,
+ &request_instant,
+ action.clone(),
+ Vec::new(),
+ crate::write::inflight_commit_metadata_bytes("INSERT", layout_two)?,
+ )
+ .await?;
+ drop(cs1);
+
+ // Group rows across batches by partition path and write one base file per partition.
+ let mut partition_rows: HashMap<String, Vec<(usize, usize)>> = HashMap::new();
+ for (batch_idx, batch) in batches.iter().enumerate() {
+ let keys = hoodie_keys_for_batch(&table.hudi_configs, batch, Some(&request_instant))?;
+ for (row_idx, key) in keys.iter().enumerate() {
+ partition_rows
+ .entry(key.partition_path.clone())
+ .or_default()
+ .push((batch_idx, row_idx));
+ }
+ }
+
+ let mut write_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ let mut files_mdt = Vec::new();
+ let mut rli_entries = Vec::new();
+ let mut stats_updates = Vec::new();
+ let mut written_paths: Vec<String> = Vec::new();
+ let mut primary_base_path = String::new();
+ let max_records = max_records_per_file(table);
+ // Monotonic auto-key row id across partitions and size-split chunks in this commit.
+ let mut auto_key_row_offset = 0usize;
+ let mut auto_key_partition_id = 0u32;
+
+ // Stable partition iteration so partition_id assignment is deterministic.
+ let mut partition_rows_sorted: Vec<_> = partition_rows.into_iter().collect();
+ partition_rows_sorted.sort_by(|a, b| a.0.cmp(&b.0));
+
+ // Plan every output file name up front and write markers before any data
+ // file is put — rollback of a crashed write is driven by these markers.
+ let mut file_plan: HashMap<String, std::collections::VecDeque<(String, String)>> =
+ HashMap::new();
+ let mut planned_markers = Vec::new();
+ for (partition_path, row_refs) in &partition_rows_sorted {
+ let chunks = row_refs.len().div_ceil(max_records).max(1);
+ let entries = file_plan.entry(partition_path.clone()).or_default();
+ for _ in 0..chunks {
+ let file_id = crate::write::new_file_id();
+ let file_name = format!("{file_id}_0-0-0_{request_instant}.parquet");
+ planned_markers.push(crate::write::markers::Marker::create(
+ partition_path,
+ &file_name,
+ ));
+ entries.push_back((file_id, file_name));
+ }
+ }
+ crate::write::markers::write_markers(storage.as_ref(), &request_instant, &planned_markers)
+ .await?;
+
+ // Phase 1 (sequential): slice + prepare every chunk, keeping table borrows
+ // out of the parallel phase.
+ struct ChunkPlan {
+ partition_path: String,
+ file_id: String,
+ file_name: String,
+ base_file_path: String,
+ row_count: i64,
+ chunk_keys: Vec<crate::index::HoodieKey>,
+ prepared: Vec<RecordBatch>,
+ }
+ let mut chunk_plans: Vec<ChunkPlan> = Vec::new();
+ for (partition_path, row_refs) in partition_rows_sorted {
+ // BTreeMap: deterministic row order in the written file.
+ let batch_indices: std::collections::BTreeMap<usize, Vec<u32>> = {
+ let mut map: std::collections::BTreeMap<usize, Vec<u32>> =
+ std::collections::BTreeMap::new();
+ for (batch_idx, row_idx) in &row_refs {
+ map.entry(*batch_idx).or_default().push(*row_idx as u32);
+ }
+ map
+ };
+ let mut partition_batches = Vec::new();
+ for (batch_idx, rows) in batch_indices {
+ partition_batches.push(take_rows(&batches[batch_idx], &rows)?);
+ }
+ let partition_batch = match partition_batches.pop() {
+ Some(only) if partition_batches.is_empty() => only,
+ Some(last) => {
+ partition_batches.push(last);
+ arrow::compute::concat_batches(&schema, &partition_batches)?
+ }
+ None => continue,
+ };
+ crate::write::ensure_partition_metadata(
+ storage.as_ref(),
+ &partition_path,
+ &request_instant,
+ )
+ .await?;
+
+ // Split oversized partitions into multiple base files (Java max.file.size).
+ let total_rows = partition_batch.num_rows();
+ let mut offset = 0usize;
+ while offset < total_rows {
+ let end = (offset + max_records).min(total_rows);
+ let indices: Vec<u32> = (offset as u32..end as u32).collect();
+ let chunk = take_rows(&partition_batch, &indices)?;
+ let chunk_row_offset = auto_key_row_offset;
+ let chunk_keys = hoodie_keys_for_batch_with_offset(
+ &table.hudi_configs,
+ &chunk,
+ Some(&request_instant),
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ auto_key_row_offset += chunk.num_rows();
+ let (file_id, file_name) = file_plan
+ .get_mut(&partition_path)
+ .and_then(std::collections::VecDeque::pop_front)
+ .ok_or_else(|| {
+ CoreError::Write(format!(
+ "file plan exhausted for partition '{partition_path}'"
+ ))
+ })?;
+ let base_file_path = relative_data_path(&partition_path, &file_name);
+ if primary_base_path.is_empty() {
+ primary_base_path = base_file_path.clone();
+ }
+ let prepared = prepare_batches_for_write_with_offset(
+ &table.hudi_configs,
+ &[chunk],
+ &request_instant,
+ &file_name,
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ chunk_plans.push(ChunkPlan {
+ partition_path: partition_path.clone(),
+ file_id,
+ file_name,
+ base_file_path,
+ row_count: indices.len() as i64,
+ chunk_keys,
+ prepared,
+ });
+ offset = end;
+ }
+ auto_key_partition_id = auto_key_partition_id.saturating_add(1);
+ }
+
+ // Phase 2 (parallel): encode + put each file on the write task pool.
+ let props = parquet_writer_props(table);
+ let collect_ranges = is_column_stats_enabled(table);
+ let tasks = chunk_plans
+ .iter()
+ .map(|plan| {
+ crate::write::parquet_file_task(
+ storage.clone(),
+ props.clone(),
+ plan.prepared.clone(),
+ plan.base_file_path.clone(),
+ collect_ranges,
+ )
+ })
+ .collect();
+ let results =
+ crate::write::run_write_tasks(tasks, crate::write::write_task_parallelism(table)).await;
+
+ // Phase 3 (sequential): assemble outputs in plan order; clean up on error.
+ let mut first_error = None;
+ for (plan, result) in chunk_plans.iter().zip(results) {
+ match result {
+ Ok(output) => {
+ written_paths.push(plan.base_file_path.clone());
+ if let Some(ranges) = output.ranges {
+ stats_updates.push(crate::write::metadata::StatsFileUpdate {
+ partition_path: plan.partition_path.clone(),
+ file_name: plan.file_name.clone(),
+ is_deleted: false,
+ ranges,
+ });
+ }
+ write_stats
+ .entry(plan.partition_path.clone())
+ .or_default()
+ .push(HoodieWriteStat {
+ file_id: Some(plan.file_id.clone()),
+ path: Some(plan.base_file_path.clone()),
+ // Basename only — FileGroup builder joins partition_path + name.
+ base_file: Some(plan.file_name.clone()),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(plan.row_count),
+ num_inserts: Some(plan.row_count),
+ total_write_bytes: Some(output.size),
+ file_size_in_bytes: Some(output.size),
+ partition_path: Some(plan.partition_path.clone()),
+ ..Default::default()
+ });
+ files_mdt.push((
+ plan.partition_path.clone(),
+ plan.file_name.clone(),
+ output.size,
+ ));
+ for key in &plan.chunk_keys {
+ rli_entries.push(RecordIndexEntry {
+ record_key: key.record_key.clone(),
+ partition_path: plan.partition_path.clone(),
+ file_id: plan.file_id.clone(),
+ instant_time_millis: crate::write::metadata::instant_to_epoch_millis_in(
+ &request_instant,
+ &crate::write::metadata::timeline_timezone(&table.hudi_configs),
+ ),
+ is_deleted: false,
+ });
+ }
+ }
+ Err(error) => {
+ if first_error.is_none() {
+ first_error = Some(error);
+ }
+ }
+ }
+ }
+ if let Some(error) = first_error {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+
+ let commit_metadata = HoodieCommitMetadata {
+ version: Some(1),
+ operation_type: Some("INSERT".to_string()),
+ partition_to_write_stats: Some(write_stats),
+ compacted: Some(false),
+ // Java HoodieCommitMetadata.extraMetadata is non-null; null → NPE in Spark.
+ extra_metadata: Some(HashMap::from([(
+ "schema".to_string(),
+ arrow_schema_to_avro_json(&schema)?,
+ )])),
+ };
+ let commit_bytes = if layout_two {
+ commit_metadata.to_avro_bytes()?
+ } else {
+ commit_metadata.to_json_bytes()?
+ };
+
+ // Crash-consistency order: data files → MDT files → MDT deltacommit →
+ // completed data commit. Finalizing the data timeline last means readers
+ // fence MDT contents by completed data instants. MDT log-file writes are
+ // "work" (unlocked); only completion runs in critical section 2.
+ let mut mdt_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ if table.is_metadata_table_enabled() {
+ let mdt_result = async {
+ mdt_stats.insert(
+ "files".to_string(),
+ update_files_partition(storage.as_ref(), &request_instant, &files_mdt).await?,
+ );
+ if is_record_index_enabled(table) {
+ let stats =
+ update_record_index(storage.as_ref(), &request_instant, &rli_entries).await?;
+ if !stats.is_empty() {
+ mdt_stats.insert("record_index".to_string(), stats);
+ }
+ }
+ if is_column_stats_enabled(table) && !stats_updates.is_empty() {
+ let stats = update_column_stats_partitions(
+ table,
+ &request_instant,
+ &stats_updates,
+ schema.as_ref(),
+ &[],
+ )
+ .await?;
+ mdt_stats.extend(stats);
+ }
+ Ok::<(), crate::error::CoreError>(())
+ }
+ .await;
+ if let Err(error) = mdt_result {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ }
+
+ // Critical section 2: complete the action (MDT deltacommit + data commit
+ // with a completion time minted under the lock), then bookkeeping.
+ let cs2 = lock.lock().await?;
+ if table.is_metadata_table_enabled()
+ && let Err(error) = crate::write::metadata::write_metadata_commit(
+ storage.as_ref(),
+ &request_instant,
+ mdt_stats,
+ )
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ let completion = if layout_two {
+ Some(generate_instant_time().await)
+ } else {
+ None
+ };
+ let instant = Instant::new_completed(request_instant.clone(), action, completion)?;
+ let commit_relative_path = instant.relative_path_with_base(&timeline_dir)?;
+ if let Err(error) = storage
+ .put_file_if_absent(&commit_relative_path, commit_bytes)
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error.into());
+ }
+ // The commit is durable from here: post-commit maintenance (markers,
+ // archival) and handle refresh are best-effort — a transient failure must
+ // not turn a committed write into an Err the caller would retry.
+ let _ =
+ crate::write::post_complete_bookkeeping(table, storage.as_ref(), &request_instant).await;
+ drop(cs2);
+
+ let _ = table.timeline.reload_completed_commits().await;
+ table.file_system_view.clear_cache();
+
+ Ok(AppendResult {
+ instant: request_instant,
+ commit_relative_path,
+ base_file_path: primary_base_path,
+ num_rows,
+ })
+}
+
+fn take_rows(batch: &RecordBatch, indices: &[u32]) -> Result<RecordBatch> {
+ let indices = UInt32Array::from(indices.to_vec());
+ let columns = batch
+ .columns()
+ .iter()
+ .map(|column| take(column.as_ref(), &indices, None))
+ .collect::<std::result::Result<Vec<_>, _>>()?;
+ RecordBatch::try_new(batch.schema(), columns).map_err(Into::into)
+}
+
+fn ensure_insert_allowed(table: &Table) -> Result<()> {
+ let strategy: String = table
+ .hudi_configs
+ .get_or_default(RecordMergeStrategy)
+ .into();
+ // Insert/append is valid for both append-only and upsertable (commit/event-time) tables.
+ match RecordMergeStrategyValue::from_str(&strategy)? {
+ RecordMergeStrategyValue::AppendOnly | RecordMergeStrategyValue::OverwriteWithLatest => {
+ Ok(())
+ }
+ }
+}
+
+fn ensure_append_only_strategy(table: &Table) -> Result<()> {
+ let strategy: String = table
+ .hudi_configs
+ .get_or_default(RecordMergeStrategy)
+ .into();
+ if RecordMergeStrategyValue::from_str(&strategy)? != RecordMergeStrategyValue::AppendOnly {
+ return Err(CoreError::Unsupported(
+ "append_only requires hoodie.record.merge.mode=append_only".to_string(),
+ ));
+ }
+ Ok(())
+}
+
+pub(crate) fn ensure_copy_on_write(table: &Table) -> Result<()> {
+ let table_type: String = table.hudi_configs.get(TableType)?.into();
+ if TableTypeValue::from_str(&table_type)? != TableTypeValue::CopyOnWrite {
+ return Err(CoreError::Unsupported(format!(
+ "append is currently only supported for COPY_ON_WRITE tables. Found '{table_type}'"
+ )));
+ }
+ Ok(())
+}
+
+fn ensure_supported_table_type(table: &Table) -> Result<()> {
+ let table_type: String = table.hudi_configs.get(TableType)?.into();
+ match TableTypeValue::from_str(&table_type)? {
+ TableTypeValue::CopyOnWrite | TableTypeValue::MergeOnRead => Ok(()),
+ }
+}
+
+pub(crate) fn is_layout_two(table: &Table) -> bool {
+ let table_version: isize = match table.hudi_configs.get(TableVersion) {
+ Ok(v) => v.into(),
+ Err(_) => return false,
+ };
+ let layout_version: isize = match table.hudi_configs.get(TimelineLayoutVersion) {
+ Ok(v) => v.into(),
+ Err(_) => return false,
+ };
+ table_version >= 8 && layout_version == 2
+}
+
+pub(crate) fn timeline_dir(table: &Table) -> String {
+ if is_layout_two(table) {
+ let timeline_path: String = table.hudi_configs.get_or_default(TimelinePath).into();
+ format!("{HUDI_METADATA_DIR}/{timeline_path}")
+ } else {
+ HUDI_METADATA_DIR.to_string()
+ }
+}
+
+/// Java `HoodieTimeGeneratorConfig` infers a 1 ms max expected clock skew when
+/// the (default) `InProcessLockProvider` is used — the single-writer case.
+const MAX_EXPECTED_CLOCK_SKEW_MS: u64 = 1;
+
+/// Skew-adjusting monotonic instant time generator (Java
+/// `SkewAdjustingTimeGenerator` + `HoodieInstantTimeGenerator.createNewInstantTime`).
+///
+/// Under a process-wide lock: capture the clock, wait out the max expected
+/// clock skew, and return the captured time — retrying (never bumping +1) until
+/// it exceeds every previously minted instant. Used for both requested and
+/// completion timestamps, so completion times are strictly greater than the
+/// requested times minted before them. Format `yyyyMMddHHmmssSSS` (UTC).
+/// Whether instants are formatted in local time (Java's
+/// `HoodieTimelineTimeZone.LOCAL`, the Hudi default) or UTC. Process-wide,
+/// mirroring `HoodieInstantTimeGenerator.setCommitTimeZone`; set from the
+/// table's `hoodie.table.timeline.timezone` when a table is created or
+/// loaded for writes. Spark writers mint LOCAL-time instants by default, so
+/// matching the table's declared zone keeps mixed-writer timelines ordered.
+static COMMIT_TIME_IS_LOCAL: std::sync::atomic::AtomicBool =
+ std::sync::atomic::AtomicBool::new(true);
+
+pub(crate) fn set_commit_timezone(timezone: &str) {
+ COMMIT_TIME_IS_LOCAL.store(
+ !timezone.eq_ignore_ascii_case("utc"),
+ std::sync::atomic::Ordering::Release,
+ );
+}
+
+pub(crate) async fn generate_instant_time() -> String {
+ static TIME_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
+ let _guard = TIME_LOCK.lock().await;
+ loop {
+ let now = Utc::now();
+ tokio::time::sleep(std::time::Duration::from_millis(MAX_EXPECTED_CLOCK_SKEW_MS)).await;
+ let candidate_millis = now.timestamp_millis();
+ // Store is race-free: all mints serialize through TIME_LOCK.
+ if candidate_millis > LAST_EPOCH_MILLIS.load(Ordering::Acquire) {
+ LAST_EPOCH_MILLIS.store(candidate_millis, Ordering::Release);
+ let format = "%Y%m%d%H%M%S%3f";
+ return if COMMIT_TIME_IS_LOCAL.load(std::sync::atomic::Ordering::Acquire) {
+ now.with_timezone(&chrono::Local).format(format).to_string()
+ } else {
+ now.format(format).to_string()
+ };
+ }
+ }
+}
+
+/// Default max base-file size (Java `hoodie.parquet.max.file.size` = 120 MiB).
+pub(crate) const DEFAULT_MAX_FILE_SIZE_BYTES: i64 = 120 * 1024 * 1024;
+/// Default record size estimate used for insert splits.
+pub(crate) const DEFAULT_RECORD_SIZE_ESTIMATE: i64 = 1024;
+
+pub(crate) fn max_records_per_file(table: &Table) -> usize {
+ let max_file_size = table
+ .hudi_configs
+ .as_options()
+ .get("hoodie.parquet.max.file.size")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_MAX_FILE_SIZE_BYTES)
+ .max(1);
+ let record_size = table
+ .hudi_configs
+ .as_options()
+ .get("hoodie.copyonwrite.record.size.estimate")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_RECORD_SIZE_ESTIMATE)
+ .max(1);
+ ((max_file_size / record_size) as usize).max(1)
+}
+
+/// Parquet `WriterProperties` resolved from `hoodie.parquet.*` configs.
+pub(crate) fn parquet_writer_props(table: &Table) -> parquet::file::properties::WriterProperties {
+ use parquet::basic::{Compression, GzipLevel, ZstdLevel};
+ use parquet::file::properties::WriterProperties;
+
+ let options = table.hudi_configs.as_options();
+ let page_size = options
+ .get("hoodie.parquet.page.size")
+ .and_then(|v| v.parse::<usize>().ok())
+ .unwrap_or(1024 * 1024)
+ .max(1);
+ let block_size = options
+ .get("hoodie.parquet.block.size")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_MAX_FILE_SIZE_BYTES)
+ .max(1);
+ let record_size = options
+ .get("hoodie.copyonwrite.record.size.estimate")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_RECORD_SIZE_ESTIMATE)
+ .max(1);
+ // parquet-rs row-group limit is row-count; approximate Java's byte block size.
+ let max_row_group_rows = ((block_size / record_size) as usize).max(1);
+ let dictionary_enabled = options
+ .get("hoodie.parquet.dictionary.enabled")
+ .map(|v| v.eq_ignore_ascii_case("true"))
+ .unwrap_or(true);
+ let codec = options
+ .get("hoodie.parquet.compression.codec")
+ .map(|s| s.as_str())
+ .unwrap_or("zstd");
+ let compression = match codec.to_ascii_lowercase().as_str() {
+ "uncompressed" | "none" => Compression::UNCOMPRESSED,
+ "snappy" => Compression::SNAPPY,
+ "gzip" | "gz" => Compression::GZIP(GzipLevel::try_new(6).unwrap_or_default()),
+ "lz4" => Compression::LZ4,
+ "brotli" => Compression::BROTLI(Default::default()),
+ // Default zstd — intentional vs Java's gzip; honor hoodie.parquet.compression.codec.
+ _ => Compression::ZSTD(ZstdLevel::try_new(3).unwrap_or_default()),
+ };
+
+ WriterProperties::builder()
+ .set_compression(compression)
+ .set_dictionary_enabled(dictionary_enabled)
+ .set_data_page_size_limit(page_size)
+ .set_dictionary_page_size_limit(page_size)
+ .set_max_row_group_row_count(Some(max_row_group_rows))
+ .build()
+}
+
+/// Encode batches to parquet bytes with pre-resolved properties (safe to run
+/// on a blocking thread with owned inputs).
+pub(crate) fn write_parquet_bytes_with_props(
+ props: parquet::file::properties::WriterProperties,
+ batches: &[RecordBatch],
+) -> Result<Vec<u8>> {
+ let cursor = Cursor::new(Vec::new());
+ let mut writer = ArrowWriter::try_new(cursor, batches[0].schema(), Some(props))?;
+ for batch in batches {
+ writer.write(batch)?;
+ }
+ let cursor = writer.into_inner()?;
+ Ok(cursor.into_inner())
+}
+
+/// Arrow→Avro schema JSON for `HoodieCommitMetadata.extraMetadata.schema`.
+///
+/// Unmapped Arrow types error out — silent `"string"` fallback corrupts Spark's
+/// TableSchemaResolver (Parquet types diverge from commit schema).
+pub(crate) fn arrow_schema_to_avro_json(schema: &arrow_schema::Schema) -> Result<String> {
+ let fields: Vec<String> = schema
+ .fields()
+ .iter()
+ .enumerate()
+ .map(|(index, f)| {
+ ensure_legal_avro_name(f.name())?;
+ // Field INDEX, not name: a name-derived path is not injective
+ // (`a.b` and `a_b` collide), and Avro refuses a redefined record.
+ let avro_type = arrow_type_to_avro_type_json_named(
+ f.data_type(),
+ &format!("hoodie_record_f{index}"),
+ )?;
+ let ty = if f.is_nullable() {
+ format!("[\"null\",{avro_type}]")
+ } else {
+ avro_type
+ };
+ Ok(format!(
+ "{{\"name\":\"{}\",\"type\":{}}}",
+ f.name().replace('\"', "\\\""),
+ ty
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+ Ok(format!(
+ "{{\"type\":\"record\",\"name\":\"hoodie_record\",\"fields\":[{}]}}",
+ fields.join(",")
+ ))
+}
+
+/// Avro field names must match [A-Za-z_][A-Za-z0-9_]*. A name that does not is
+/// rejected rather than renamed: renaming would diverge from the parquet column
+/// and silently mislabel the data, while an unparseable commit schema makes the
+/// table unreadable by every engine (including this one).
+fn ensure_legal_avro_name(name: &str) -> Result<()> {
+ let mut chars = name.chars();
+ let legal = chars
+ .next()
+ .is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
+ && chars.all(|c| c.is_ascii_alphanumeric() || c == '_');
+ if legal {
+ return Ok(());
+ }
+ Err(CoreError::Unsupported(format!(
+ "column name '{name}' is not a legal Avro name ([A-Za-z_][A-Za-z0-9_]*), \
+ so it cannot be written into Hudi commit metadata"
+ )))
+}
+
+/// `path` makes every nested Avro record name unique: Avro forbids redefining
+/// a named type, so a fixed name breaks any schema with two structs — Java's
+/// `Schema.Parser` rejects the whole commit schema.
+fn arrow_type_to_avro_type_json_named(dt: &arrow_schema::DataType, path: &str) -> Result<String> {
+ use arrow_schema::DataType;
+ match dt {
+ DataType::Boolean => Ok("\"boolean\"".to_string()),
+ DataType::Int8 | DataType::Int16 | DataType::Int32 => Ok("\"int\"".to_string()),
+ DataType::Int64 => Ok("\"long\"".to_string()),
+ DataType::Float32 => Ok("\"float\"".to_string()),
+ DataType::Float64 => Ok("\"double\"".to_string()),
+ DataType::Utf8 | DataType::LargeUtf8 => {
+ Ok("{\"type\":\"string\",\"avro.java.string\":\"String\"}".to_string())
+ }
+ DataType::Binary | DataType::LargeBinary => Ok("\"bytes\"".to_string()),
+ DataType::Date32 => Ok("{\"type\":\"int\",\"logicalType\":\"date\"}".to_string()),
+ DataType::Timestamp(unit, tz) => {
+ // A timezone-less Arrow timestamp is Spark's TimestampNTZType,
+ // which maps to Avro local-timestamp-*; mapping it to the
+ // UTC-adjusted logical type shifts values on read.
+ let local = tz.is_none();
+ let logical = match unit {
+ arrow_schema::TimeUnit::Millisecond if local => "local-timestamp-millis",
+ arrow_schema::TimeUnit::Microsecond if local => "local-timestamp-micros",
+ arrow_schema::TimeUnit::Millisecond => "timestamp-millis",
+ arrow_schema::TimeUnit::Microsecond => "timestamp-micros",
+ other => {
+ return Err(CoreError::Unsupported(format!(
+ "commit schema does not support Arrow timestamp unit {other:?}"
+ )));
+ }
+ };
+ Ok(format!(
+ "{{\"type\":\"long\",\"logicalType\":\"{logical}\"}}"
+ ))
+ }
+ DataType::Decimal128(precision, scale) | DataType::Decimal256(precision, scale) => {
+ Ok(format!(
+ "{{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":{precision},\"scale\":{scale}}}"
+ ))
+ }
+ DataType::List(field) | DataType::LargeList(field) => {
+ let item =
+ arrow_type_to_avro_type_json_named(field.data_type(), &format!("{path}_item"))?;
+ let item = if field.is_nullable() {
+ format!("[\"null\",{item}]")
+ } else {
+ item
+ };
+ Ok(format!("{{\"type\":\"array\",\"items\":{item}}}"))
+ }
+ DataType::Struct(fields) => {
+ let nested: Vec<String> = fields
+ .iter()
+ .enumerate()
+ .map(|(index, f)| {
+ ensure_legal_avro_name(f.name())?;
+ let avro_type = arrow_type_to_avro_type_json_named(
+ f.data_type(),
+ &format!("{path}f{index}"),
+ )?;
+ let ty = if f.is_nullable() {
+ format!("[\"null\",{avro_type}]")
+ } else {
+ avro_type
+ };
+ Ok(format!(
+ "{{\"name\":\"{}\",\"type\":{}}}",
+ f.name().replace('\"', "\\\""),
+ ty
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+ Ok(format!(
+ "{{\"type\":\"record\",\"name\":\"{path}_record\",\"fields\":[{}]}}",
+ nested.join(",")
+ ))
+ }
+ other => Err(CoreError::Unsupported(format!(
+ "cannot map Arrow type {other:?} to Avro for commit metadata schema"
+ ))),
+ }
+}
+
+async fn ensure_append_schema_matches_table(
+ table: &Table,
+ batch_schema: &arrow_schema::Schema,
+) -> Result<()> {
+ if table.timeline.completed_commits.is_empty() {
+ return Ok(());
+ }
+ let table_schema = table.get_schema().await?;
+ let batch_data = strip_meta_fields_from_schema(batch_schema);
+ if schemas_equal_by_fields(&table_schema, &batch_data) {
+ return Ok(());
+ }
+ Err(CoreError::Write(format!(
+ "append batch schema does not match table schema (table fields={:?}, batch fields={:?})",
+ table_schema
+ .fields()
+ .iter()
+ .map(|f| (f.name().clone(), f.data_type().clone()))
+ .collect::<Vec<_>>(),
+ batch_data
+ .fields()
+ .iter()
+ .map(|f| (f.name().clone(), f.data_type().clone()))
+ .collect::<Vec<_>>(),
+ )))
+}
+
+pub(crate) fn strip_meta_fields_from_schema(schema: &arrow_schema::Schema) -> arrow_schema::Schema {
+ let fields: Vec<_> = schema
+ .fields()
+ .iter()
+ .filter(|f| !f.name().starts_with("_hoodie_"))
+ .cloned()
+ .collect();
+ arrow_schema::Schema::new(fields)
+}
+
+/// Prepend the Hudi meta fields to a data schema when absent — written files
+/// always carry them, so index bookkeeping must consider them.
+pub(crate) fn schema_with_meta_fields(schema: &arrow_schema::Schema) -> arrow_schema::Schema {
+ // Match the meta fields by exact name: a data column may legitimately
+ // begin with `_hoodie_` (Hudi's own `_hoodie_is_deleted` marker does), and
+ // treating it as proof the meta fields are present would silently skip
+ // them.
+ let meta: std::collections::HashSet<&str> =
+ crate::metadata::meta_field::MetaField::field_names()
+ .into_iter()
+ .collect();
+ if schema
+ .fields()
+ .iter()
+ .any(|f| meta.contains(f.name().as_str()))
+ {
+ return schema.clone();
+ }
+ let mut fields: Vec<arrow_schema::FieldRef> =
+ crate::metadata::meta_field::MetaField::field_names()
+ .into_iter()
+ .map(|name| {
+ std::sync::Arc::new(arrow_schema::Field::new(
+ name,
+ arrow_schema::DataType::Utf8,
+ true,
+ ))
+ })
+ .collect();
+ fields.extend(schema.fields().iter().cloned());
+ arrow_schema::Schema::new(fields)
+}
+
+fn schemas_equal_by_fields(a: &arrow_schema::Schema, b: &arrow_schema::Schema) -> bool {
+ if a.fields().len() != b.fields().len() {
+ return false;
+ }
+ a.fields()
+ .iter()
+ .zip(b.fields().iter())
+ .all(|(fa, fb)| fa.name() == fb.name() && fa.data_type() == fb.data_type())
+}
+
+pub(crate) fn prepare_batches_for_write(
+ hudi_configs: &crate::config::HudiConfigs,
+ batches: &[RecordBatch],
+ instant: &str,
+ file_name: &str,
+) -> Result<Vec<RecordBatch>> {
+ prepare_batches_for_write_with_offset(hudi_configs, batches, instant, file_name, 0, 0)
+}
+
+pub(crate) fn prepare_batches_for_write_with_offset(
+ hudi_configs: &crate::config::HudiConfigs,
+ batches: &[RecordBatch],
+ instant: &str,
+ file_name: &str,
+ row_id_offset: usize,
+ partition_id: u32,
+) -> Result<Vec<RecordBatch>> {
+ let populates_meta_fields: bool = hudi_configs.get_or_default(PopulatesMetaFields).into();
+ if !populates_meta_fields {
+ return Ok(batches.to_vec());
+ }
+
+ batches
Review Comment:
The offset isn't advanced across batches here, so a multi-batch `overwrite` / `dynamic_partition_overwrite` on an auto-keyed table reuses `{instant}_0_0`, `{instant}_0_1`, ... per batch, while the RLI is built with fresh offsets. I pushed [e05904e1](https://github.com/apache/hudi-rs/pull/666/commits/e05904e1b8013923764baa9f9cf408889ae7f91f), which carries the offset across batches so stored keys are unique and match the RLI.
##########
crates/core/src/table/mod.rs:
##########
@@ -201,17 +200,27 @@ fn instant_time_minus_one(instant_time: &str) -> String {
}
impl Table {
- /// Get or initialize the cached metadata table instance.
+ /// Open a fresh metadata-table handle from storage (no instance cache).
///
- /// Returns `Ok(&Table)` if metadata table is successfully created or was already cached.
- /// The instance is created once and reused across all subsequent calls.
- pub(crate) async fn get_or_init_metadata_table(&self) -> Result<&Table> {
- self.cached_metadata_table
- .get_or_try_init(|| async {
- log::debug!("Initializing cached metadata table instance");
- self.new_metadata_table().await
- })
- .await
+ /// Writers and readers reload the data timeline first, then open MDT relative
+ /// to that view. Commit locking / conflict checks are deferred to a later
+ /// concurrency-control project.
+ pub(crate) async fn get_or_init_metadata_table(&self) -> Result<Table> {
Review Comment:
With the cached MDT handle removed, each `get_file_slices` re-opens the metadata table (properties, timeline, and file listing), which adds round trips to every DataFusion/Python read. I pushed [2971ba78](https://github.com/apache/hudi-rs/pull/666/commits/2971ba78cce1a2915a263570c5f6aa781d568597), which restores the cache for readers and resets it wherever a write changes the handle's timeline.
##########
crates/core/src/write/keygen.rs:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Derive Hoodie record keys and hive-style partition paths from write batches.
+
+use arrow::array::{Array, StringArray};
+use arrow::record_batch::RecordBatch;
+
+use crate::Result;
+use crate::config::HudiConfigs;
+use crate::config::table::HudiTableConfig::{
+ IsHiveStylePartitioning, PartitionFields, RecordKeyFields,
+};
+use crate::error::CoreError;
+use crate::index::HoodieKey;
+
+/// Build per-row [`HoodieKey`]s from a write batch.
+///
+/// When `hoodie.table.recordkey.fields` is unset/empty, keys are auto-generated
+/// like Java `AutoRecordGenWrapper*KeyGenerator`: `{instant}_{partitionId}_{rowId}`.
+///
+/// `row_id_offset` continues the auto-key row counter across size-split chunks and
+/// partitions within one commit. `partition_id` distinguishes writer partitions
+/// (Java `HoodieRecord.generateSequenceId`).
+///
+/// Partition paths use hive-style (`field=value/...`) when
+/// `hoodie.datasource.write.hive_style_partitioning` is true; otherwise value-only
+/// segments. Null partition values are rejected. Partition columns are not stripped
+/// from the batch — callers must leave them in the written Parquet.
+/// Placeholder instant for pre-fence validation passes: [`hoodie_keys_for_batch`]
+/// uses the instant only to FORMAT auto-generated keys, never to decide errors,
+/// so deriving keys with this dummy value surfaces every deterministic input
+/// error (missing/null/wrong-type key or partition columns) before an instant
+/// is requested.
+pub(crate) const VALIDATION_INSTANT: &str = "00000000000000000";
+
+/// Run key/partition derivation for its error side only — call before fencing
+/// an instant so invalid input leaves no timeline trace.
+pub(crate) fn validate_keygen_inputs(
+ hudi_configs: &HudiConfigs,
+ batches: &[RecordBatch],
+) -> Result<()> {
+ for batch in batches {
+ hoodie_keys_for_batch(hudi_configs, batch, Some(VALIDATION_INSTANT))?;
+ }
+ Ok(())
+}
+
+pub fn hoodie_keys_for_batch(
+ hudi_configs: &HudiConfigs,
+ batch: &RecordBatch,
+ instant: Option<&str>,
+) -> Result<Vec<HoodieKey>> {
+ hoodie_keys_for_batch_with_offset(hudi_configs, batch, instant, 0, 0)
+}
+
+/// Like [`hoodie_keys_for_batch`], with an explicit auto-key row offset and partition id.
+pub fn hoodie_keys_for_batch_with_offset(
+ hudi_configs: &HudiConfigs,
+ batch: &RecordBatch,
+ instant: Option<&str>,
+ row_id_offset: usize,
+ partition_id: u32,
+) -> Result<Vec<HoodieKey>> {
+ let record_key_fields: Option<Vec<String>> =
+ hudi_configs.try_get(RecordKeyFields)?.map(|v| v.into());
+ let auto_keys = record_key_fields
+ .as_ref()
+ .map(|f| f.is_empty() || (f.len() == 1 && f[0].is_empty()))
+ .unwrap_or(true);
+
+ let partition_fields: Vec<String> = hudi_configs.get_or_default(PartitionFields).into();
+ let hive_style: bool = hudi_configs.get_or_default(IsHiveStylePartitioning).into();
+
+ if auto_keys {
+ let instant = instant.ok_or_else(|| {
+ CoreError::Write(
+ "auto record-key generation requires a commit instant timestamp".to_string(),
+ )
+ })?;
+ let mut keys = Vec::with_capacity(batch.num_rows());
+ for row in 0..batch.num_rows() {
+ // Match Java HoodieRecord.generateSequenceId(instant, partitionId, rowId).
+ let record_key = format!("{instant}_{partition_id}_{}", row_id_offset + row);
+ let partition_path = partition_path_for_row(batch, row, &partition_fields, hive_style)?;
+ keys.push(HoodieKey {
+ record_key,
+ partition_path,
+ });
+ }
+ return Ok(keys);
+ }
+
+ let Some(fields) = record_key_fields else {
+ // Unreachable: auto_keys above covers None, but the no-panic rule
+ // wants an error, not an unwrap.
+ return Err(CoreError::Config(
+ crate::config::error::ConfigError::NotFound("record key fields".to_string()),
+ ));
+ };
+ if fields.len() != 1 {
+ return Err(CoreError::Unsupported(
+ "writes currently require exactly one record key field (or none for auto keys)"
+ .to_string(),
+ ));
+ }
+ let key_name = &fields[0];
+ let key_array = batch
+ .column_by_name(key_name)
+ .ok_or_else(|| {
+ CoreError::Schema(format!(
+ "record key field '{key_name}' is missing from write batch"
+ ))
+ })?
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .ok_or_else(|| {
+ CoreError::Unsupported(format!(
+ "record key field '{key_name}' must be Utf8 for writes"
+ ))
+ })?;
+
+ let mut keys = Vec::with_capacity(batch.num_rows());
+ for row in 0..batch.num_rows() {
+ if key_array.is_null(row) {
+ return Err(CoreError::Write(
+ "record key values must be non-null".to_string(),
+ ));
+ }
+ let record_key = key_array.value(row).to_string();
+ let partition_path = partition_path_for_row(batch, row, &partition_fields, hive_style)?;
+ keys.push(HoodieKey {
+ record_key,
+ partition_path,
+ });
+ }
+ Ok(keys)
+}
+
+fn partition_path_for_row(
Review Comment:
Writes ignored the table's key generator and `partitionpath.urlencode`, so tables using them were silently mis-keyed (e.g. `2024/01/02` with urlencode=true became three directory levels), and an empty partition value landed at the table root. I pushed [3e551abc](https://github.com/apache/hudi-rs/pull/666/commits/3e551abc29f42257158adce23026910b5727e472), which resolves the key generator like Java, ports `escapePathName`, maps empty/null partition values to `__HIVE_DEFAULT_PARTITION__`, and fails fast on generators it can't reproduce, with follow-ups [e3e76bcc](https://github.com/apache/hudi-rs/pull/666/commits/e3e76bcc42154ea0b6835230ef1e2d0f58f32d27), [af6ad6cc](https://github.com/apache/hudi-rs/pull/666/commits/af6ad6cc39184c03c4d8160b96e54a167a544886), [92e24534](https://github.com/apache/hudi-rs/pull/666/commits/92e24534d9d9993b33643e8379e4d0fc54484075) and [905d461f](https://github.com/apache/hudi-rs/pull/666/commits/905d461fccd34e35674efdda2bcf85e77e3f7f1a) for Java's tv<9 c
omplex-keygen validation, guarding only key-deriving writes (with typed partition fields still refused for SET), and key-column delete filters.
##########
crates/core/src/write/archival.rs:
##########
@@ -0,0 +1,562 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Timeline archival into the LSM history timeline (Java `TimelineArchiverV2`
+//! + `LSMTimelineWriter`).
+//!
+//! When the active timeline holds more than `hoodie.keep.max.commits`
+//! completed commits, the oldest are archived down to `hoodie.keep.min.commits`:
+//! written as one level-0 parquet of `HoodieLSMTimelineInstant` rows under
+//! `{timeline}/history/`, registered in a new `manifest_{N}` (JSON) pointed at
+//! by `_version_`, then removed from the active timeline (pending fencing
+//! files first, completed files after — Java's ordering). Runs inside the
+//! completion critical section (user design; Java runs it post-commit).
+//!
+//! LSM level compaction (merging L0 files into L1+) is not yet implemented —
+//! readers are unaffected since every file stays manifest-listed.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use arrow_array::{Array, ArrayRef, BinaryArray, Int32Array, RecordBatch, StringArray};
+use arrow_schema::{DataType, Field, Schema};
+use serde::{Deserialize, Serialize};
+
+use crate::Result;
+use crate::error::CoreError;
+use crate::storage::Storage;
+
+/// Java `hoodie.keep.min.commits` default.
+pub(crate) const DEFAULT_KEEP_MIN_COMMITS: usize = 20;
+/// Java `hoodie.keep.max.commits` default.
+pub(crate) const DEFAULT_KEEP_MAX_COMMITS: usize = 30;
+/// Java `hoodie.timeline.manifest.retained.versions` default.
+const RETAINED_MANIFEST_VERSIONS: i64 = 3;
+
+/// Java `HoodieLSMTimelineManifest`: `{"files":[{"fileName":..,"fileLen":..}]}`.
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+struct LsmManifest {
+ files: Vec<LsmFileEntry>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+struct LsmFileEntry {
+ file_name: String,
+ file_len: i64,
+}
+
+#[derive(Debug, Clone)]
+struct CompletedInstant {
+ requested: String,
+ completion: String,
+ action: String,
+ file_name: String,
+}
+
+/// Archive the timeline at `timeline_dir` if it exceeds `max_keep` completed
+/// commits, retaining the newest `min_keep`.
+pub(crate) async fn archive_timeline_if_needed(
+ storage: &Storage,
+ timeline_dir: &str,
+ min_keep: usize,
+ max_keep: usize,
+) -> Result<()> {
+ let listed = match storage.list_files(Some(timeline_dir)).await {
+ Ok(files) => files,
+ Err(crate::storage::error::StorageError::ObjectStoreError(
+ object_store::Error::NotFound { .. },
+ )) => return Ok(()),
+ Err(error) => return Err(error.into()),
+ };
+
+ let mut completed: Vec<CompletedInstant> = Vec::new();
+ let mut fencing_by_instant: HashMap<String, Vec<String>> = HashMap::new();
+ let mut earliest_pending: Option<String> = None;
+ for file in &listed {
+ let name = &file.name;
+ let Some(first) = name.chars().next() else {
+ continue;
+ };
+ if !first.is_ascii_digit() {
+ continue;
+ }
+ let requested: String = name.chars().take_while(char::is_ascii_digit).collect();
+ if name.ends_with(".requested") || name.ends_with(".inflight") {
+ fencing_by_instant
+ .entry(requested.clone())
+ .or_default()
+ .push(name.clone());
+ continue;
+ }
+ // Completed layout-v2 name: `{req}_{comp}.{action}`.
+ let Some((stem, action)) = name.rsplit_once('.') else {
+ continue;
+ };
+ let Some((req, comp)) = stem.split_once('_') else {
+ continue;
+ };
+ completed.push(CompletedInstant {
+ requested: req.to_string(),
+ completion: comp.to_string(),
+ action: action.to_string(),
+ file_name: name.clone(),
+ });
+ let _ = requested;
+ }
+ // Pending = fenced without a completed file.
+ let completed_set: std::collections::HashSet<&str> =
+ completed.iter().map(|c| c.requested.as_str()).collect();
+ for ts in fencing_by_instant.keys() {
+ if !completed_set.contains(ts.as_str()) && earliest_pending.as_ref().is_none_or(|e| ts < e)
+ {
+ earliest_pending = Some(ts.clone());
+ }
+ }
+
+ completed.sort_by(|a, b| a.requested.cmp(&b.requested));
+ // A replacecommit's partitionToReplaceFileIds is what hides the file
+ // groups it replaced; until a cleaner physically deletes those groups,
+ // archiving it resurrects their rows. Java blocks archival at the first
+ // uncleaned replace instant; with no cleaner implemented, that means
+ // never archiving at or past the earliest active replacecommit.
+ let earliest_replace = completed
+ .iter()
+ .filter(|c| c.action == "replacecommit")
+ .map(|c| c.requested.clone())
+ .min();
+ // A savepoint pins its commit and everything after it: archiving past one
+ // would discard the very files the savepoint exists to retain (Java's
+ // TimelineArchiverV2 stops at the first savepoint unless
+ // hoodie.archive.beyond.savepoint, default false).
+ let earliest_savepoint = completed
+ .iter()
+ .filter(|c| c.action == "savepoint")
+ .map(|c| c.requested.clone())
+ .min();
+ let earliest_pending = [earliest_pending, earliest_replace, earliest_savepoint]
+ .into_iter()
+ .flatten()
+ .min();
+ let is_commit_action =
+ |action: &str| matches!(action, "commit" | "deltacommit" | "replacecommit");
+ let commit_count = completed
+ .iter()
+ .filter(|c| is_commit_action(&c.action))
+ .count();
+ // Misconfigured min > max would underflow below and archive the whole
+ // timeline; treat min as the effective floor.
+ let min_keep = min_keep.min(max_keep);
+ if commit_count <= max_keep {
+ return Ok(());
+ }
+
+ // Archive the oldest commits down to `min_keep`, but never at or past the
+ // earliest pending instant; sweep along non-commit actions (rollbacks) up
+ // to the newest archived commit.
+ let mut commits_to_archive = commit_count - min_keep;
+ let mut max_archived_commit: Option<String> = None;
+ let mut to_archive: Vec<CompletedInstant> = Vec::new();
Review Comment:
Java's archiver keeps the last completed commit before the earliest pending instant, and the reader's `is_committed` boundary relies on that; here a pending Spark compaction or clustering plan that sits while rs makes 20+ commits gets every commit below it archived, so its inflight output reads as committed. I pushed [91d541cd](https://github.com/apache/hudi-rs/pull/666/commits/91d541cdf32cd7546fd0a85945e8780bda233bb0) to match TimelineArchiverV2's retention rule, and [bb646de7](https://github.com/apache/hudi-rs/pull/666/commits/bb646de77b060c9df099c06a8ab996810881101a) so a failed fencing-file delete stops archival instead of leaving an archived commit looking like a failed write.
##########
crates/core/src/index/record.rs:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Record-level index backed by the metadata table `record_index` partition.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use crate::Result;
+use crate::error::CoreError;
+use crate::file_group::log_file::scanner::{LogFileScanner, ScanResult};
+use crate::hfile::{HFileReader, HFileRecord};
+use crate::index::{HoodieIndex, HoodieKey, RecordLocation};
+use crate::metadata::table::encode::decode_record_index_entry;
+use crate::metadata::table::records::MetadataPartitionType;
+use crate::storage::Storage;
+use crate::table::Table;
+use crate::timeline::selector::InstantRange;
+use crate::write::metadata::epoch_millis_to_instant;
+
+/// Metadata-table record-level index (global: key = record key).
+#[derive(Clone, Debug, Default)]
+pub struct RecordIndex;
+
+impl HoodieIndex for RecordIndex {
+ async fn tag_location(
+ &self,
+ table: &Table,
+ keys: &[HoodieKey],
+ ) -> Result<HashMap<HoodieKey, Option<RecordLocation>>> {
+ let mut locations = keys
+ .iter()
+ .cloned()
+ .map(|key| (key, None))
+ .collect::<HashMap<_, _>>();
+ if keys.is_empty() {
+ return Ok(locations);
+ }
+
+ let storage = table.file_system_view.storage.clone();
+ let merged = load_record_index_map(table, storage).await?;
+ for key in keys {
+ if let Some(entry) = merged.get(&key.record_key) {
+ locations.insert(
+ key.clone(),
+ Some(RecordLocation {
+ file_id: entry.file_id.clone(),
+ instant_time: epoch_millis_to_instant(entry.instant_time_millis),
+ partition_path: entry.partition_path.clone(),
+ }),
+ );
+ }
+ }
+ Ok(locations)
+ }
+}
+
+struct LoadedLocation {
+ file_id: String,
+ partition_path: String,
+ instant_time_millis: i64,
+}
+
+async fn load_record_index_map(
+ table: &Table,
+ storage: Arc<Storage>,
+) -> Result<HashMap<String, LoadedLocation>> {
+ let partition = MetadataPartitionType::RecordIndex.partition_name();
+ // Slice-resolved: only each file group's LIVE slice (multi-base aware
+ // after a Spark MDT compaction), logs in version order.
+ let mut base_paths = Vec::new();
+ let mut log_paths = Vec::new();
+ for (_, base, logs) in
+ crate::write::metadata::mdt_partition_latest_slices(storage.as_ref(), partition).await?
+ {
+ base_paths.push(base);
+ log_paths.extend(logs);
+ }
+
+ let mut merged: HashMap<String, Option<LoadedLocation>> = HashMap::new();
+ for base in base_paths {
+ let mut reader = HFileReader::open(storage.as_ref(), &base)
+ .await
+ .map_err(|e| CoreError::HFile(format!("failed to open record_index base: {e:?}")))?;
+ let records = reader
+ .collect_records()
+ .map_err(|e| CoreError::HFile(format!("failed to read record_index base: {e:?}")))?;
+ apply_hfile_records(&mut merged, &records)?;
+ }
+ if !log_paths.is_empty() {
+ // Only trust log blocks whose instant completed on the data timeline
+ // (Java getValidInstantTimestamps) — orphan MDT commits are skipped.
+ let valid = table.valid_metadata_instants().await?;
+ let scanner = LogFileScanner::new(table.hudi_configs.clone(), storage);
+ let range = InstantRange::exact_match(valid, "UTC");
+ match scanner.scan(log_paths, &range).await? {
+ ScanResult::HFileRecords(records) => apply_hfile_records(&mut merged, &records)?,
+ ScanResult::Empty => {}
+ ScanResult::RecordBatches(_) => {
+ return Err(CoreError::MetadataTable(
+ "record_index logs must contain HFile data blocks".to_string(),
Review Comment:
After a Spark MDT compaction, record_index logs that hold only delete blocks (e.g. from an rs MOR `delete_keys`) make this error out, so every later upsert or delete fails until a put lands. I pushed [308f7238](https://github.com/apache/hudi-rs/pull/666/commits/308f723847a8e6ed32042da9a585580ac4f8259e), which applies delete-only logs as tombstones like Java's log record scanner.
##########
crates/core/src/write/append.rs:
##########
@@ -0,0 +1,1227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::collections::HashMap;
+use std::io::Cursor;
+use std::str::FromStr;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicI64, Ordering};
+
+use arrow::array::{ArrayRef, StringArray, UInt32Array};
+use arrow::record_batch::RecordBatch;
+use arrow_select::take::take;
+use chrono::Utc;
+use parquet::arrow::ArrowWriter;
+
+use crate::Result;
+use crate::config::table::HudiTableConfig::{
+ PopulatesMetaFields, RecordMergeStrategy, TableType, TableVersion, TimelineLayoutVersion,
+ TimelinePath,
+};
+use crate::config::table::TableTypeValue;
+use crate::error::CoreError;
+use crate::index::is_record_index_enabled;
+use crate::merge::RecordMergeStrategyValue;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::RecordIndexEntry;
+use crate::table::Table;
+use crate::timeline::instant::{Action, Instant};
+use crate::write::keygen::{
+ hoodie_keys_for_batch, hoodie_keys_for_batch_with_offset, relative_data_path,
+};
+use crate::write::metadata::{
+ is_column_stats_enabled, update_column_stats_partitions, update_files_partition,
+ update_record_index,
+};
+
+/// Result of an append write.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct AppendResult {
+ pub instant: String,
+ pub commit_relative_path: String,
+ pub base_file_path: String,
+ pub num_rows: usize,
+}
+
+static LAST_EPOCH_MILLIS: AtomicI64 = AtomicI64::new(0);
+
+/// Append one or more record batches as a single INSERT commit.
+pub async fn append_batches(table: &mut Table, batches: &[RecordBatch]) -> Result<AppendResult> {
+ append_batches_inner(table, batches, false).await
+}
+
+/// Append one or more record batches, requiring append-only merge mode.
+pub async fn append_batches_only(
+ table: &mut Table,
+ batches: &[RecordBatch],
+) -> Result<AppendResult> {
+ append_batches_inner(table, batches, true).await
+}
+
+async fn append_batches_inner(
+ table: &mut Table,
+ batches: &[RecordBatch],
+ require_append_only: bool,
+) -> Result<AppendResult> {
+ table.reload_timeline_for_write().await?;
+ if !table.is_mor() {
+ if require_append_only {
+ ensure_append_only_strategy(table)?;
+ } else {
+ ensure_insert_allowed(table)?;
+ }
+ }
+ ensure_supported_table_type(table)?;
+
+ if batches.is_empty() {
+ return Err(CoreError::Write(
+ "append requires at least one RecordBatch".to_string(),
+ ));
+ }
+ let schema = batches[0].schema();
+ for batch in batches.iter().skip(1) {
+ if batch.schema().as_ref() != schema.as_ref() {
+ return Err(CoreError::Write(
+ "All RecordBatches in append must share the same schema".to_string(),
+ ));
+ }
+ }
+ ensure_append_schema_matches_table(table, schema.as_ref()).await?;
+ // Must be expressible in commit metadata (Avro JSON) before fencing.
+ arrow_schema_to_avro_json(&strip_meta_fields_from_schema(schema.as_ref()))?;
+ let num_rows: usize = batches.iter().map(|b| b.num_rows()).sum();
+ if num_rows == 0 {
+ return Err(CoreError::Write(
+ "append requires at least one row".to_string(),
+ ));
+ }
+
+ crate::write::keygen::validate_keygen_inputs(&table.hudi_configs, batches)?;
+
+ let layout_two = is_layout_two(table);
+ let action = if table.is_mor() {
+ Action::DeltaCommit
+ } else {
+ Action::Commit
+ };
+ let timeline_dir = timeline_dir(table);
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ // Critical section 1: mint the instant and fence it, then work unlocked.
+ let cs1 = lock.lock().await?;
+ let request_instant = generate_instant_time().await;
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir,
+ &request_instant,
+ action.clone(),
+ Vec::new(),
+ crate::write::inflight_commit_metadata_bytes("INSERT", layout_two)?,
+ )
+ .await?;
+ drop(cs1);
+
+ // Group rows across batches by partition path and write one base file per partition.
+ let mut partition_rows: HashMap<String, Vec<(usize, usize)>> = HashMap::new();
+ for (batch_idx, batch) in batches.iter().enumerate() {
+ let keys = hoodie_keys_for_batch(&table.hudi_configs, batch, Some(&request_instant))?;
+ for (row_idx, key) in keys.iter().enumerate() {
+ partition_rows
+ .entry(key.partition_path.clone())
+ .or_default()
+ .push((batch_idx, row_idx));
+ }
+ }
+
+ let mut write_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ let mut files_mdt = Vec::new();
+ let mut rli_entries = Vec::new();
+ let mut stats_updates = Vec::new();
+ let mut written_paths: Vec<String> = Vec::new();
+ let mut primary_base_path = String::new();
+ let max_records = max_records_per_file(table);
+ // Monotonic auto-key row id across partitions and size-split chunks in this commit.
+ let mut auto_key_row_offset = 0usize;
+ let mut auto_key_partition_id = 0u32;
+
+ // Stable partition iteration so partition_id assignment is deterministic.
+ let mut partition_rows_sorted: Vec<_> = partition_rows.into_iter().collect();
+ partition_rows_sorted.sort_by(|a, b| a.0.cmp(&b.0));
+
+ // Plan every output file name up front and write markers before any data
+ // file is put — rollback of a crashed write is driven by these markers.
+ let mut file_plan: HashMap<String, std::collections::VecDeque<(String, String)>> =
+ HashMap::new();
+ let mut planned_markers = Vec::new();
+ for (partition_path, row_refs) in &partition_rows_sorted {
+ let chunks = row_refs.len().div_ceil(max_records).max(1);
+ let entries = file_plan.entry(partition_path.clone()).or_default();
+ for _ in 0..chunks {
+ let file_id = crate::write::new_file_id();
+ let file_name = format!("{file_id}_0-0-0_{request_instant}.parquet");
+ planned_markers.push(crate::write::markers::Marker::create(
+ partition_path,
+ &file_name,
+ ));
+ entries.push_back((file_id, file_name));
+ }
+ }
+ crate::write::markers::write_markers(storage.as_ref(), &request_instant, &planned_markers)
+ .await?;
+
+ // Phase 1 (sequential): slice + prepare every chunk, keeping table borrows
+ // out of the parallel phase.
+ struct ChunkPlan {
+ partition_path: String,
+ file_id: String,
+ file_name: String,
+ base_file_path: String,
+ row_count: i64,
+ chunk_keys: Vec<crate::index::HoodieKey>,
+ prepared: Vec<RecordBatch>,
+ }
+ let mut chunk_plans: Vec<ChunkPlan> = Vec::new();
+ for (partition_path, row_refs) in partition_rows_sorted {
+ // BTreeMap: deterministic row order in the written file.
+ let batch_indices: std::collections::BTreeMap<usize, Vec<u32>> = {
+ let mut map: std::collections::BTreeMap<usize, Vec<u32>> =
+ std::collections::BTreeMap::new();
+ for (batch_idx, row_idx) in &row_refs {
+ map.entry(*batch_idx).or_default().push(*row_idx as u32);
+ }
+ map
+ };
+ let mut partition_batches = Vec::new();
+ for (batch_idx, rows) in batch_indices {
+ partition_batches.push(take_rows(&batches[batch_idx], &rows)?);
+ }
+ let partition_batch = match partition_batches.pop() {
+ Some(only) if partition_batches.is_empty() => only,
+ Some(last) => {
+ partition_batches.push(last);
+ arrow::compute::concat_batches(&schema, &partition_batches)?
+ }
+ None => continue,
+ };
+ crate::write::ensure_partition_metadata(
+ storage.as_ref(),
+ &partition_path,
+ &request_instant,
+ )
+ .await?;
+
+ // Split oversized partitions into multiple base files (Java max.file.size).
+ let total_rows = partition_batch.num_rows();
+ let mut offset = 0usize;
+ while offset < total_rows {
+ let end = (offset + max_records).min(total_rows);
+ let indices: Vec<u32> = (offset as u32..end as u32).collect();
+ let chunk = take_rows(&partition_batch, &indices)?;
+ let chunk_row_offset = auto_key_row_offset;
+ let chunk_keys = hoodie_keys_for_batch_with_offset(
+ &table.hudi_configs,
+ &chunk,
+ Some(&request_instant),
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ auto_key_row_offset += chunk.num_rows();
+ let (file_id, file_name) = file_plan
+ .get_mut(&partition_path)
+ .and_then(std::collections::VecDeque::pop_front)
+ .ok_or_else(|| {
+ CoreError::Write(format!(
+ "file plan exhausted for partition '{partition_path}'"
+ ))
+ })?;
+ let base_file_path = relative_data_path(&partition_path, &file_name);
+ if primary_base_path.is_empty() {
+ primary_base_path = base_file_path.clone();
+ }
+ let prepared = prepare_batches_for_write_with_offset(
+ &table.hudi_configs,
+ &[chunk],
+ &request_instant,
+ &file_name,
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ chunk_plans.push(ChunkPlan {
+ partition_path: partition_path.clone(),
+ file_id,
+ file_name,
+ base_file_path,
+ row_count: indices.len() as i64,
+ chunk_keys,
+ prepared,
+ });
+ offset = end;
+ }
+ auto_key_partition_id = auto_key_partition_id.saturating_add(1);
+ }
+
+ // Phase 2 (parallel): encode + put each file on the write task pool.
+ let props = parquet_writer_props(table);
+ let collect_ranges = is_column_stats_enabled(table);
+ let tasks = chunk_plans
+ .iter()
+ .map(|plan| {
+ crate::write::parquet_file_task(
+ storage.clone(),
+ props.clone(),
+ plan.prepared.clone(),
+ plan.base_file_path.clone(),
+ collect_ranges,
+ )
+ })
+ .collect();
+ let results =
+ crate::write::run_write_tasks(tasks, crate::write::write_task_parallelism(table)).await;
+
+ // Phase 3 (sequential): assemble outputs in plan order; clean up on error.
+ let mut first_error = None;
+ for (plan, result) in chunk_plans.iter().zip(results) {
+ match result {
+ Ok(output) => {
+ written_paths.push(plan.base_file_path.clone());
+ if let Some(ranges) = output.ranges {
+ stats_updates.push(crate::write::metadata::StatsFileUpdate {
+ partition_path: plan.partition_path.clone(),
+ file_name: plan.file_name.clone(),
+ is_deleted: false,
+ ranges,
+ });
+ }
+ write_stats
+ .entry(plan.partition_path.clone())
+ .or_default()
+ .push(HoodieWriteStat {
+ file_id: Some(plan.file_id.clone()),
+ path: Some(plan.base_file_path.clone()),
+ // Basename only — FileGroup builder joins partition_path + name.
+ base_file: Some(plan.file_name.clone()),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(plan.row_count),
+ num_inserts: Some(plan.row_count),
+ total_write_bytes: Some(output.size),
+ file_size_in_bytes: Some(output.size),
+ partition_path: Some(plan.partition_path.clone()),
+ ..Default::default()
+ });
+ files_mdt.push((
+ plan.partition_path.clone(),
+ plan.file_name.clone(),
+ output.size,
+ ));
+ for key in &plan.chunk_keys {
+ rli_entries.push(RecordIndexEntry {
+ record_key: key.record_key.clone(),
+ partition_path: plan.partition_path.clone(),
+ file_id: plan.file_id.clone(),
+ instant_time_millis: crate::write::metadata::instant_to_epoch_millis_in(
+ &request_instant,
+ &crate::write::metadata::timeline_timezone(&table.hudi_configs),
+ ),
+ is_deleted: false,
+ });
+ }
+ }
+ Err(error) => {
+ if first_error.is_none() {
+ first_error = Some(error);
+ }
+ }
+ }
+ }
+ if let Some(error) = first_error {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+
+ let commit_metadata = HoodieCommitMetadata {
+ version: Some(1),
+ operation_type: Some("INSERT".to_string()),
+ partition_to_write_stats: Some(write_stats),
+ compacted: Some(false),
+ // Java HoodieCommitMetadata.extraMetadata is non-null; null → NPE in Spark.
+ extra_metadata: Some(HashMap::from([(
+ "schema".to_string(),
+ arrow_schema_to_avro_json(&schema)?,
+ )])),
+ };
+ let commit_bytes = if layout_two {
+ commit_metadata.to_avro_bytes()?
+ } else {
+ commit_metadata.to_json_bytes()?
+ };
+
+ // Crash-consistency order: data files → MDT files → MDT deltacommit →
+ // completed data commit. Finalizing the data timeline last means readers
+ // fence MDT contents by completed data instants. MDT log-file writes are
+ // "work" (unlocked); only completion runs in critical section 2.
+ let mut mdt_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ if table.is_metadata_table_enabled() {
+ let mdt_result = async {
+ mdt_stats.insert(
+ "files".to_string(),
+ update_files_partition(storage.as_ref(), &request_instant, &files_mdt).await?,
+ );
+ if is_record_index_enabled(table) {
+ let stats =
+ update_record_index(storage.as_ref(), &request_instant, &rli_entries).await?;
+ if !stats.is_empty() {
+ mdt_stats.insert("record_index".to_string(), stats);
+ }
+ }
+ if is_column_stats_enabled(table) && !stats_updates.is_empty() {
+ let stats = update_column_stats_partitions(
+ table,
+ &request_instant,
+ &stats_updates,
+ schema.as_ref(),
+ &[],
+ )
+ .await?;
+ mdt_stats.extend(stats);
+ }
+ Ok::<(), crate::error::CoreError>(())
+ }
+ .await;
+ if let Err(error) = mdt_result {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ }
+
+ // Critical section 2: complete the action (MDT deltacommit + data commit
+ // with a completion time minted under the lock), then bookkeeping.
+ let cs2 = lock.lock().await?;
+ if table.is_metadata_table_enabled()
+ && let Err(error) = crate::write::metadata::write_metadata_commit(
+ storage.as_ref(),
+ &request_instant,
+ mdt_stats,
+ )
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ let completion = if layout_two {
+ Some(generate_instant_time().await)
+ } else {
+ None
+ };
+ let instant = Instant::new_completed(request_instant.clone(), action, completion)?;
+ let commit_relative_path = instant.relative_path_with_base(&timeline_dir)?;
+ if let Err(error) = storage
+ .put_file_if_absent(&commit_relative_path, commit_bytes)
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error.into());
+ }
+ // The commit is durable from here: post-commit maintenance (markers,
+ // archival) and handle refresh are best-effort — a transient failure must
+ // not turn a committed write into an Err the caller would retry.
+ let _ =
+ crate::write::post_complete_bookkeeping(table, storage.as_ref(), &request_instant).await;
+ drop(cs2);
+
+ let _ = table.timeline.reload_completed_commits().await;
+ table.file_system_view.clear_cache();
+
+ Ok(AppendResult {
+ instant: request_instant,
+ commit_relative_path,
+ base_file_path: primary_base_path,
+ num_rows,
+ })
+}
+
+fn take_rows(batch: &RecordBatch, indices: &[u32]) -> Result<RecordBatch> {
+ let indices = UInt32Array::from(indices.to_vec());
+ let columns = batch
+ .columns()
+ .iter()
+ .map(|column| take(column.as_ref(), &indices, None))
+ .collect::<std::result::Result<Vec<_>, _>>()?;
+ RecordBatch::try_new(batch.schema(), columns).map_err(Into::into)
+}
+
+fn ensure_insert_allowed(table: &Table) -> Result<()> {
+ let strategy: String = table
+ .hudi_configs
+ .get_or_default(RecordMergeStrategy)
+ .into();
+ // Insert/append is valid for both append-only and upsertable (commit/event-time) tables.
+ match RecordMergeStrategyValue::from_str(&strategy)? {
+ RecordMergeStrategyValue::AppendOnly | RecordMergeStrategyValue::OverwriteWithLatest => {
+ Ok(())
+ }
+ }
+}
+
+fn ensure_append_only_strategy(table: &Table) -> Result<()> {
+ let strategy: String = table
+ .hudi_configs
+ .get_or_default(RecordMergeStrategy)
+ .into();
+ if RecordMergeStrategyValue::from_str(&strategy)? != RecordMergeStrategyValue::AppendOnly {
+ return Err(CoreError::Unsupported(
+ "append_only requires hoodie.record.merge.mode=append_only".to_string(),
+ ));
+ }
+ Ok(())
+}
+
+pub(crate) fn ensure_copy_on_write(table: &Table) -> Result<()> {
+ let table_type: String = table.hudi_configs.get(TableType)?.into();
+ if TableTypeValue::from_str(&table_type)? != TableTypeValue::CopyOnWrite {
+ return Err(CoreError::Unsupported(format!(
+ "append is currently only supported for COPY_ON_WRITE tables. Found '{table_type}'"
+ )));
+ }
+ Ok(())
+}
+
+fn ensure_supported_table_type(table: &Table) -> Result<()> {
+ let table_type: String = table.hudi_configs.get(TableType)?.into();
+ match TableTypeValue::from_str(&table_type)? {
+ TableTypeValue::CopyOnWrite | TableTypeValue::MergeOnRead => Ok(()),
+ }
+}
+
+pub(crate) fn is_layout_two(table: &Table) -> bool {
+ let table_version: isize = match table.hudi_configs.get(TableVersion) {
+ Ok(v) => v.into(),
+ Err(_) => return false,
+ };
+ let layout_version: isize = match table.hudi_configs.get(TimelineLayoutVersion) {
+ Ok(v) => v.into(),
+ Err(_) => return false,
+ };
+ table_version >= 8 && layout_version == 2
+}
+
+pub(crate) fn timeline_dir(table: &Table) -> String {
+ if is_layout_two(table) {
+ let timeline_path: String = table.hudi_configs.get_or_default(TimelinePath).into();
+ format!("{HUDI_METADATA_DIR}/{timeline_path}")
+ } else {
+ HUDI_METADATA_DIR.to_string()
+ }
+}
+
+/// Java `HoodieTimeGeneratorConfig` infers a 1 ms max expected clock skew when
+/// the (default) `InProcessLockProvider` is used — the single-writer case.
+const MAX_EXPECTED_CLOCK_SKEW_MS: u64 = 1;
+
+/// Skew-adjusting monotonic instant time generator (Java
+/// `SkewAdjustingTimeGenerator` + `HoodieInstantTimeGenerator.createNewInstantTime`).
+///
+/// Under a process-wide lock: capture the clock, wait out the max expected
+/// clock skew, and return the captured time — retrying (never bumping +1) until
+/// it exceeds every previously minted instant. Used for both requested and
+/// completion timestamps, so completion times are strictly greater than the
+/// requested times minted before them. Format `yyyyMMddHHmmssSSS` (UTC).
+/// Whether instants are formatted in local time (Java's
+/// `HoodieTimelineTimeZone.LOCAL`, the Hudi default) or UTC. Process-wide,
+/// mirroring `HoodieInstantTimeGenerator.setCommitTimeZone`; set from the
+/// table's `hoodie.table.timeline.timezone` when a table is created or
+/// loaded for writes. Spark writers mint LOCAL-time instants by default, so
+/// matching the table's declared zone keeps mixed-writer timelines ordered.
+static COMMIT_TIME_IS_LOCAL: std::sync::atomic::AtomicBool =
+ std::sync::atomic::AtomicBool::new(true);
+
+pub(crate) fn set_commit_timezone(timezone: &str) {
+ COMMIT_TIME_IS_LOCAL.store(
+ !timezone.eq_ignore_ascii_case("utc"),
+ std::sync::atomic::Ordering::Release,
+ );
+}
+
+pub(crate) async fn generate_instant_time() -> String {
+ static TIME_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
+ let _guard = TIME_LOCK.lock().await;
+ loop {
+ let now = Utc::now();
+ tokio::time::sleep(std::time::Duration::from_millis(MAX_EXPECTED_CLOCK_SKEW_MS)).await;
+ let candidate_millis = now.timestamp_millis();
+ // Store is race-free: all mints serialize through TIME_LOCK.
+ if candidate_millis > LAST_EPOCH_MILLIS.load(Ordering::Acquire) {
+ LAST_EPOCH_MILLIS.store(candidate_millis, Ordering::Release);
+ let format = "%Y%m%d%H%M%S%3f";
+ return if COMMIT_TIME_IS_LOCAL.load(std::sync::atomic::Ordering::Acquire) {
+ now.with_timezone(&chrono::Local).format(format).to_string()
+ } else {
+ now.format(format).to_string()
+ };
+ }
+ }
+}
+
+/// Default max base-file size (Java `hoodie.parquet.max.file.size` = 120 MiB).
+pub(crate) const DEFAULT_MAX_FILE_SIZE_BYTES: i64 = 120 * 1024 * 1024;
+/// Default record size estimate used for insert splits.
+pub(crate) const DEFAULT_RECORD_SIZE_ESTIMATE: i64 = 1024;
+
+pub(crate) fn max_records_per_file(table: &Table) -> usize {
+ let max_file_size = table
+ .hudi_configs
+ .as_options()
+ .get("hoodie.parquet.max.file.size")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_MAX_FILE_SIZE_BYTES)
+ .max(1);
+ let record_size = table
+ .hudi_configs
+ .as_options()
+ .get("hoodie.copyonwrite.record.size.estimate")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_RECORD_SIZE_ESTIMATE)
+ .max(1);
+ ((max_file_size / record_size) as usize).max(1)
+}
+
+/// Parquet `WriterProperties` resolved from `hoodie.parquet.*` configs.
+pub(crate) fn parquet_writer_props(table: &Table) -> parquet::file::properties::WriterProperties {
+ use parquet::basic::{Compression, GzipLevel, ZstdLevel};
+ use parquet::file::properties::WriterProperties;
+
+ let options = table.hudi_configs.as_options();
+ let page_size = options
+ .get("hoodie.parquet.page.size")
+ .and_then(|v| v.parse::<usize>().ok())
+ .unwrap_or(1024 * 1024)
+ .max(1);
+ let block_size = options
+ .get("hoodie.parquet.block.size")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_MAX_FILE_SIZE_BYTES)
+ .max(1);
+ let record_size = options
+ .get("hoodie.copyonwrite.record.size.estimate")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_RECORD_SIZE_ESTIMATE)
+ .max(1);
+ // parquet-rs row-group limit is row-count; approximate Java's byte block size.
+ let max_row_group_rows = ((block_size / record_size) as usize).max(1);
+ let dictionary_enabled = options
+ .get("hoodie.parquet.dictionary.enabled")
+ .map(|v| v.eq_ignore_ascii_case("true"))
+ .unwrap_or(true);
+ let codec = options
+ .get("hoodie.parquet.compression.codec")
+ .map(|s| s.as_str())
+ .unwrap_or("zstd");
+ let compression = match codec.to_ascii_lowercase().as_str() {
+ "uncompressed" | "none" => Compression::UNCOMPRESSED,
+ "snappy" => Compression::SNAPPY,
+ "gzip" | "gz" => Compression::GZIP(GzipLevel::try_new(6).unwrap_or_default()),
+ "lz4" => Compression::LZ4,
+ "brotli" => Compression::BROTLI(Default::default()),
+ // Default zstd — intentional vs Java's gzip; honor hoodie.parquet.compression.codec.
+ _ => Compression::ZSTD(ZstdLevel::try_new(3).unwrap_or_default()),
+ };
+
+ WriterProperties::builder()
+ .set_compression(compression)
+ .set_dictionary_enabled(dictionary_enabled)
+ .set_data_page_size_limit(page_size)
+ .set_dictionary_page_size_limit(page_size)
+ .set_max_row_group_row_count(Some(max_row_group_rows))
+ .build()
+}
+
+/// Encode batches to parquet bytes with pre-resolved properties (safe to run
+/// on a blocking thread with owned inputs).
+pub(crate) fn write_parquet_bytes_with_props(
+ props: parquet::file::properties::WriterProperties,
+ batches: &[RecordBatch],
+) -> Result<Vec<u8>> {
+ let cursor = Cursor::new(Vec::new());
+ let mut writer = ArrowWriter::try_new(cursor, batches[0].schema(), Some(props))?;
+ for batch in batches {
+ writer.write(batch)?;
+ }
+ let cursor = writer.into_inner()?;
+ Ok(cursor.into_inner())
+}
+
+/// Arrow→Avro schema JSON for `HoodieCommitMetadata.extraMetadata.schema`.
+///
+/// Unmapped Arrow types error out — silent `"string"` fallback corrupts Spark's
+/// TableSchemaResolver (Parquet types diverge from commit schema).
+pub(crate) fn arrow_schema_to_avro_json(schema: &arrow_schema::Schema) -> Result<String> {
+ let fields: Vec<String> = schema
+ .fields()
+ .iter()
+ .enumerate()
+ .map(|(index, f)| {
+ ensure_legal_avro_name(f.name())?;
+ // Field INDEX, not name: a name-derived path is not injective
+ // (`a.b` and `a_b` collide), and Avro refuses a redefined record.
+ let avro_type = arrow_type_to_avro_type_json_named(
+ f.data_type(),
+ &format!("hoodie_record_f{index}"),
+ )?;
+ let ty = if f.is_nullable() {
+ format!("[\"null\",{avro_type}]")
+ } else {
+ avro_type
+ };
+ Ok(format!(
+ "{{\"name\":\"{}\",\"type\":{}}}",
+ f.name().replace('\"', "\\\""),
+ ty
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+ Ok(format!(
+ "{{\"type\":\"record\",\"name\":\"hoodie_record\",\"fields\":[{}]}}",
+ fields.join(",")
+ ))
+}
+
+/// Avro field names must match [A-Za-z_][A-Za-z0-9_]*. A name that does not is
+/// rejected rather than renamed: renaming would diverge from the parquet column
+/// and silently mislabel the data, while an unparseable commit schema makes the
+/// table unreadable by every engine (including this one).
+fn ensure_legal_avro_name(name: &str) -> Result<()> {
+ let mut chars = name.chars();
+ let legal = chars
+ .next()
+ .is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
+ && chars.all(|c| c.is_ascii_alphanumeric() || c == '_');
+ if legal {
+ return Ok(());
+ }
+ Err(CoreError::Unsupported(format!(
+ "column name '{name}' is not a legal Avro name ([A-Za-z_][A-Za-z0-9_]*), \
+ so it cannot be written into Hudi commit metadata"
+ )))
+}
+
+/// `path` makes every nested Avro record name unique: Avro forbids redefining
+/// a named type, so a fixed name breaks any schema with two structs — Java's
+/// `Schema.Parser` rejects the whole commit schema.
+fn arrow_type_to_avro_type_json_named(dt: &arrow_schema::DataType, path: &str) -> Result<String> {
+ use arrow_schema::DataType;
+ match dt {
+ DataType::Boolean => Ok("\"boolean\"".to_string()),
+ DataType::Int8 | DataType::Int16 | DataType::Int32 => Ok("\"int\"".to_string()),
+ DataType::Int64 => Ok("\"long\"".to_string()),
+ DataType::Float32 => Ok("\"float\"".to_string()),
+ DataType::Float64 => Ok("\"double\"".to_string()),
+ DataType::Utf8 | DataType::LargeUtf8 => {
+ Ok("{\"type\":\"string\",\"avro.java.string\":\"String\"}".to_string())
+ }
+ DataType::Binary | DataType::LargeBinary => Ok("\"bytes\"".to_string()),
+ DataType::Date32 => Ok("{\"type\":\"int\",\"logicalType\":\"date\"}".to_string()),
+ DataType::Timestamp(unit, tz) => {
+ // A timezone-less Arrow timestamp is Spark's TimestampNTZType,
+ // which maps to Avro local-timestamp-*; mapping it to the
+ // UTC-adjusted logical type shifts values on read.
+ let local = tz.is_none();
+ let logical = match unit {
+ arrow_schema::TimeUnit::Millisecond if local => "local-timestamp-millis",
+ arrow_schema::TimeUnit::Microsecond if local => "local-timestamp-micros",
+ arrow_schema::TimeUnit::Millisecond => "timestamp-millis",
+ arrow_schema::TimeUnit::Microsecond => "timestamp-micros",
+ other => {
+ return Err(CoreError::Unsupported(format!(
+ "commit schema does not support Arrow timestamp unit {other:?}"
+ )));
+ }
+ };
+ Ok(format!(
+ "{{\"type\":\"long\",\"logicalType\":\"{logical}\"}}"
+ ))
+ }
+ DataType::Decimal128(precision, scale) | DataType::Decimal256(precision, scale) => {
+ Ok(format!(
+ "{{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":{precision},\"scale\":{scale}}}"
+ ))
+ }
+ DataType::List(field) | DataType::LargeList(field) => {
+ let item =
+ arrow_type_to_avro_type_json_named(field.data_type(), &format!("{path}_item"))?;
+ let item = if field.is_nullable() {
+ format!("[\"null\",{item}]")
+ } else {
+ item
+ };
+ Ok(format!("{{\"type\":\"array\",\"items\":{item}}}"))
+ }
+ DataType::Struct(fields) => {
+ let nested: Vec<String> = fields
+ .iter()
+ .enumerate()
+ .map(|(index, f)| {
+ ensure_legal_avro_name(f.name())?;
+ let avro_type = arrow_type_to_avro_type_json_named(
+ f.data_type(),
+ &format!("{path}f{index}"),
+ )?;
+ let ty = if f.is_nullable() {
+ format!("[\"null\",{avro_type}]")
+ } else {
+ avro_type
+ };
+ Ok(format!(
+ "{{\"name\":\"{}\",\"type\":{}}}",
+ f.name().replace('\"', "\\\""),
+ ty
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+ Ok(format!(
+ "{{\"type\":\"record\",\"name\":\"{path}_record\",\"fields\":[{}]}}",
+ nested.join(",")
+ ))
+ }
+ other => Err(CoreError::Unsupported(format!(
+ "cannot map Arrow type {other:?} to Avro for commit metadata schema"
+ ))),
+ }
+}
+
+async fn ensure_append_schema_matches_table(
+ table: &Table,
+ batch_schema: &arrow_schema::Schema,
+) -> Result<()> {
+ if table.timeline.completed_commits.is_empty() {
+ return Ok(());
+ }
+ let table_schema = table.get_schema().await?;
+ let batch_data = strip_meta_fields_from_schema(batch_schema);
+ if schemas_equal_by_fields(&table_schema, &batch_data) {
+ return Ok(());
+ }
+ Err(CoreError::Write(format!(
+ "append batch schema does not match table schema (table fields={:?}, batch fields={:?})",
+ table_schema
+ .fields()
+ .iter()
+ .map(|f| (f.name().clone(), f.data_type().clone()))
+ .collect::<Vec<_>>(),
+ batch_data
+ .fields()
+ .iter()
+ .map(|f| (f.name().clone(), f.data_type().clone()))
+ .collect::<Vec<_>>(),
+ )))
+}
+
+pub(crate) fn strip_meta_fields_from_schema(schema: &arrow_schema::Schema) -> arrow_schema::Schema {
+ let fields: Vec<_> = schema
+ .fields()
+ .iter()
+ .filter(|f| !f.name().starts_with("_hoodie_"))
Review Comment:
`_hoodie_is_deleted` is user data rather than a meta field, but the `_hoodie_` prefix check strips it, so the column drops out of the commit schema, a second append with the same schema fails the match, and a COW upsert with the flag set keeps the row. I pushed [79c7503c](https://github.com/apache/hudi-rs/pull/666/commits/79c7503ce036befeb1ad4d08eca54245b476158c) and [e922b7cd](https://github.com/apache/hudi-rs/pull/666/commits/e922b7cd63e6a7dec737c6f6a6ad35de6a4bcea3), which match Java's exact meta-field names and honor the marker on COW upsert and update, dropping the RLI entry.
##########
crates/core/src/write/rollback.rs:
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Eager marker-based rollback of failed writes (Java `EAGER` cleaning policy).
+//!
+//! At write start, any pending data instant (fenced but never completed) is
+//! rolled back: its marker-listed base/log files are deleted (tv8+ rollback is
+//! deletion-only), its completed-but-orphaned MDT deltacommit and MDT log
+//! files are removed, a `rollback` instant records the operation, and the
+//! failed instant's fencing files and marker dir are cleaned up.
+
+use std::collections::HashMap;
+
+use crate::Result;
+use crate::metadata::rollback::{
+ HoodieInstantInfo, HoodieRollbackMetadata, HoodieRollbackPartitionMetadata, HoodieRollbackPlan,
+ HoodieRollbackRequest,
+};
+use crate::storage::Storage;
+use crate::table::Table;
+use crate::write::append::{generate_instant_time, is_layout_two, timeline_dir};
+use crate::write::markers::{delete_marker_dir, parse_marker_name, read_markers};
+
+const METADATA_BASE: &str = ".hoodie/metadata";
+
+/// A pending (fenced, never completed) instant on the data timeline.
+#[derive(Debug, Clone)]
+struct PendingInstant {
+ timestamp: String,
+ action: String,
+ /// Fencing file names present on the timeline (to delete after rollback).
+ fencing_files: Vec<String>,
+}
+
+/// Roll back all pending data instants (Java `rollbackFailedWrites`, EAGER).
+///
+/// Single-writer: any pending instant belongs to a crashed previous write.
+/// Rollback instants themselves and MDT-internal instants are never targets.
+pub(crate) async fn rollback_failed_writes(table: &Table) -> Result<()> {
+ let storage = table.file_system_view.storage.clone();
+ let timeline = timeline_dir(table);
+ let pending = find_pending_instants(storage.as_ref(), &timeline).await?;
+ for instant in pending {
+ rollback_instant(table, storage.as_ref(), &timeline, &instant).await?;
+ }
+ Ok(())
+}
+
+/// The action infix of a fencing file name: `{ts}.{action}.requested|inflight`
+/// or the bare COW forms `{ts}.inflight` / `{ts}.requested` (empty action).
+///
+/// Must not depend on which of an instant's fencing files is listed first —
+/// directory listing order is filesystem-specific (ext4 returns hash order,
+/// so `.inflight` can precede `.commit.requested`).
+fn action_from_fencing_name(name: &str, timestamp: &str) -> String {
+ let rest = name.trim_start_matches(timestamp).trim_start_matches('.');
+ match rest {
+ "inflight" | "requested" => String::new(),
+ other => other
+ .trim_end_matches(".requested")
+ .trim_end_matches(".inflight")
+ .to_string(),
+ }
+}
+
+async fn find_pending_instants(
+ storage: &Storage,
+ timeline_dir: &str,
+) -> Result<Vec<PendingInstant>> {
+ let files = match storage.list_files(Some(timeline_dir)).await {
+ Ok(files) => files,
+ Err(crate::storage::error::StorageError::ObjectStoreError(
+ object_store::Error::NotFound { .. },
+ )) => return Ok(Vec::new()),
+ Err(error) => return Err(error.into()),
+ };
+ let mut completed = std::collections::HashSet::new();
+ let mut fenced: HashMap<String, (String, Vec<String>)> = HashMap::new();
+ for file in &files {
+ let name = &file.name;
+ let Some(first) = name.chars().next() else {
+ continue;
+ };
+ if !first.is_ascii_digit() {
+ continue;
+ }
+ let timestamp: String = name.chars().take_while(char::is_ascii_digit).collect();
+ if name.ends_with(".requested") || name.ends_with(".inflight") {
+ let action = action_from_fencing_name(name, ×tamp);
+ let entry = fenced
+ .entry(timestamp)
+ .or_insert_with(|| (action.clone(), Vec::new()));
+ // Prefer the action-bearing name (COW inflight has no action infix).
+ if entry.0.is_empty() && !action.is_empty() {
+ entry.0 = action;
+ }
+ entry.1.push(name.clone());
+ } else {
+ completed.insert(timestamp);
+ }
+ }
+ let mut pending: Vec<PendingInstant> = fenced
+ .into_iter()
+ .filter(|(ts, _)| !completed.contains(ts))
+ // Rollback/compaction-style instants are never rolled back here.
+ .filter(|(_, (action, _))| {
+ matches!(
+ action.as_str(),
+ "commit" | "deltacommit" | "replacecommit" | ""
+ )
+ })
+ .map(|(timestamp, (action, fencing_files))| PendingInstant {
+ timestamp,
+ action: if action.is_empty() {
+ "commit".to_string()
+ } else {
+ action
+ },
+ fencing_files,
+ })
+ .collect();
+ pending.sort_by(|a, b| a.timestamp.cmp(&b.timestamp));
+ Ok(pending)
+}
+
+async fn rollback_instant(
+ table: &Table,
+ storage: &Storage,
+ timeline_dir: &str,
+ pending: &PendingInstant,
+) -> Result<()> {
+ let start_time = std::time::Instant::now();
+ let failed_ts = &pending.timestamp;
+
+ // Markers tell us exactly what the failed write intended to put. A crash
+ // before markers means no data or MDT files were written (markers are
+ // written first), so an empty list still produces a valid rollback.
+ let markers = read_markers(storage, failed_ts).await?;
+ let mut requests_by_partition: HashMap<String, HoodieRollbackRequest> = HashMap::new();
+ let mut partition_meta: HashMap<String, HoodieRollbackPartitionMetadata> = HashMap::new();
+ let mut total_deleted = 0i32;
+ for marker in &markers {
+ let Some((file_path, _io_type)) = parse_marker_name(marker) else {
+ continue;
+ };
+ let (partition, file_name) = match file_path.rsplit_once('/') {
+ Some((partition, name)) => (partition.to_string(), name.to_string()),
+ None => (String::new(), file_path.clone()),
+ };
+ let is_log_file = file_name.starts_with('.') && file_name.contains(".log.");
+ let request = requests_by_partition
+ .entry(partition.clone())
+ .or_insert_with(|| HoodieRollbackRequest {
+ partition_path: partition.clone(),
+ file_id: None,
+ latest_base_instant: None,
+ files_to_be_deleted: Vec::new(),
+ log_blocks_to_be_deleted: None,
+ });
+ if is_log_file {
+ request
+ .log_blocks_to_be_deleted
+ .get_or_insert_with(HashMap::new)
+ .insert(file_path.clone(), 1);
+ } else {
+ request.files_to_be_deleted.push(file_path.clone());
+ }
+ let deleted = storage.delete_file(&file_path).await.is_ok();
+ let meta = partition_meta.entry(partition.clone()).or_insert_with(|| {
+ HoodieRollbackPartitionMetadata {
+ partition_path: partition.clone(),
+ ..Default::default()
+ }
+ });
+ if deleted {
+ total_deleted += 1;
+ meta.success_delete_files.push(file_path);
+ } else {
+ meta.failed_delete_files.push(file_path);
Review Comment:
If a delete fails here the rollback still completes and removes the fencing files, so once archival passes that timestamp the leftover file reads as committed (the orphan MDT cleanup ignores errors the same way). I pushed [1ef2dbbc](https://github.com/apache/hudi-rs/pull/666/commits/1ef2dbbc27560315251593f20d507227648f3bdb) and [ddb892db](https://github.com/apache/hudi-rs/pull/666/commits/ddb892dba2dee1a78b2c765760a30a24546f22f9), which fail the rollback when a file it must delete survives or an MDT partition cannot be listed, so it stays pending and retries like Java's RollbackHelper.
##########
crates/core/src/table/mod.rs:
##########
@@ -201,17 +200,27 @@ fn instant_time_minus_one(instant_time: &str) -> String {
}
impl Table {
- /// Get or initialize the cached metadata table instance.
+ /// Open a fresh metadata-table handle from storage (no instance cache).
///
- /// Returns `Ok(&Table)` if metadata table is successfully created or was already cached.
- /// The instance is created once and reused across all subsequent calls.
- pub(crate) async fn get_or_init_metadata_table(&self) -> Result<&Table> {
- self.cached_metadata_table
- .get_or_try_init(|| async {
- log::debug!("Initializing cached metadata table instance");
- self.new_metadata_table().await
- })
- .await
+ /// Writers and readers reload the data timeline first, then open MDT relative
+ /// to that view. Commit locking / conflict checks are deferred to a later
+ /// concurrency-control project.
+ pub(crate) async fn get_or_init_metadata_table(&self) -> Result<Table> {
+ self.new_metadata_table().await
+ }
+
+ /// Reload the active timeline and drop listing caches before a write plans.
+ pub(crate) async fn reload_timeline_for_write(&mut self) -> Result<()> {
+ crate::write::ensure_writable_table_version(self)?;
+ // Instants must be minted in the table's declared timeline timezone
+ // (Spark writers default to LOCAL); set before any instant is minted.
+ crate::write::set_commit_timezone(&self.timezone());
+ // Eager rollback of crashed writes (Java rollbackFailedWrites) before
+ // any new instant is minted.
+ crate::write::rollback::rollback_failed_writes(self).await?;
Review Comment:
Eager rollback runs outside the lock and treats every pending instant as failed, so a second `Table` handle writing to the same path in-process rolls back a write that is still running, which then completes over deleted files. Since single-writer is the scope here, I treated it as a known limitation and pushed [18a0e1a6](https://github.com/apache/hudi-rs/pull/666/commits/18a0e1a69675c5506c093d72aae96bf297c73e95), which states in `lock.rs` and the design doc that a table supports one writer handle at a time.
##########
crates/core/src/write/rollback.rs:
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Eager marker-based rollback of failed writes (Java `EAGER` cleaning policy).
+//!
+//! At write start, any pending data instant (fenced but never completed) is
+//! rolled back: its marker-listed base/log files are deleted (tv8+ rollback is
+//! deletion-only), its completed-but-orphaned MDT deltacommit and MDT log
+//! files are removed, a `rollback` instant records the operation, and the
+//! failed instant's fencing files and marker dir are cleaned up.
+
+use std::collections::HashMap;
+
+use crate::Result;
+use crate::metadata::rollback::{
+ HoodieInstantInfo, HoodieRollbackMetadata, HoodieRollbackPartitionMetadata, HoodieRollbackPlan,
+ HoodieRollbackRequest,
+};
+use crate::storage::Storage;
+use crate::table::Table;
+use crate::write::append::{generate_instant_time, is_layout_two, timeline_dir};
+use crate::write::markers::{delete_marker_dir, parse_marker_name, read_markers};
+
+const METADATA_BASE: &str = ".hoodie/metadata";
+
+/// A pending (fenced, never completed) instant on the data timeline.
+#[derive(Debug, Clone)]
+struct PendingInstant {
+ timestamp: String,
+ action: String,
+ /// Fencing file names present on the timeline (to delete after rollback).
+ fencing_files: Vec<String>,
+}
+
+/// Roll back all pending data instants (Java `rollbackFailedWrites`, EAGER).
+///
+/// Single-writer: any pending instant belongs to a crashed previous write.
+/// Rollback instants themselves and MDT-internal instants are never targets.
+pub(crate) async fn rollback_failed_writes(table: &Table) -> Result<()> {
+ let storage = table.file_system_view.storage.clone();
+ let timeline = timeline_dir(table);
+ let pending = find_pending_instants(storage.as_ref(), &timeline).await?;
+ for instant in pending {
+ rollback_instant(table, storage.as_ref(), &timeline, &instant).await?;
+ }
+ Ok(())
+}
+
+/// The action infix of a fencing file name: `{ts}.{action}.requested|inflight`
+/// or the bare COW forms `{ts}.inflight` / `{ts}.requested` (empty action).
+///
+/// Must not depend on which of an instant's fencing files is listed first —
+/// directory listing order is filesystem-specific (ext4 returns hash order,
+/// so `.inflight` can precede `.commit.requested`).
+fn action_from_fencing_name(name: &str, timestamp: &str) -> String {
+ let rest = name.trim_start_matches(timestamp).trim_start_matches('.');
+ match rest {
+ "inflight" | "requested" => String::new(),
+ other => other
+ .trim_end_matches(".requested")
+ .trim_end_matches(".inflight")
+ .to_string(),
+ }
+}
+
+async fn find_pending_instants(
+ storage: &Storage,
+ timeline_dir: &str,
+) -> Result<Vec<PendingInstant>> {
+ let files = match storage.list_files(Some(timeline_dir)).await {
+ Ok(files) => files,
+ Err(crate::storage::error::StorageError::ObjectStoreError(
+ object_store::Error::NotFound { .. },
+ )) => return Ok(Vec::new()),
+ Err(error) => return Err(error.into()),
+ };
+ let mut completed = std::collections::HashSet::new();
+ let mut fenced: HashMap<String, (String, Vec<String>)> = HashMap::new();
+ for file in &files {
+ let name = &file.name;
+ let Some(first) = name.chars().next() else {
+ continue;
+ };
+ if !first.is_ascii_digit() {
+ continue;
+ }
+ let timestamp: String = name.chars().take_while(char::is_ascii_digit).collect();
+ if name.ends_with(".requested") || name.ends_with(".inflight") {
+ let action = action_from_fencing_name(name, ×tamp);
+ let entry = fenced
+ .entry(timestamp)
+ .or_insert_with(|| (action.clone(), Vec::new()));
+ // Prefer the action-bearing name (COW inflight has no action infix).
+ if entry.0.is_empty() && !action.is_empty() {
+ entry.0 = action;
+ }
+ entry.1.push(name.clone());
+ } else {
+ completed.insert(timestamp);
+ }
+ }
+ let mut pending: Vec<PendingInstant> = fenced
+ .into_iter()
+ .filter(|(ts, _)| !completed.contains(ts))
+ // Rollback/compaction-style instants are never rolled back here.
+ .filter(|(_, (action, _))| {
+ matches!(
+ action.as_str(),
+ "commit" | "deltacommit" | "replacecommit" | ""
+ )
+ })
+ .map(|(timestamp, (action, fencing_files))| PendingInstant {
+ timestamp,
+ action: if action.is_empty() {
+ "commit".to_string()
+ } else {
+ action
+ },
+ fencing_files,
+ })
+ .collect();
+ pending.sort_by(|a, b| a.timestamp.cmp(&b.timestamp));
+ Ok(pending)
+}
+
+async fn rollback_instant(
+ table: &Table,
+ storage: &Storage,
+ timeline_dir: &str,
+ pending: &PendingInstant,
+) -> Result<()> {
+ let start_time = std::time::Instant::now();
+ let failed_ts = &pending.timestamp;
+
+ // Markers tell us exactly what the failed write intended to put. A crash
+ // before markers means no data or MDT files were written (markers are
+ // written first), so an empty list still produces a valid rollback.
+ let markers = read_markers(storage, failed_ts).await?;
+ let mut requests_by_partition: HashMap<String, HoodieRollbackRequest> = HashMap::new();
+ let mut partition_meta: HashMap<String, HoodieRollbackPartitionMetadata> = HashMap::new();
+ let mut total_deleted = 0i32;
+ for marker in &markers {
+ let Some((file_path, _io_type)) = parse_marker_name(marker) else {
+ continue;
+ };
+ let (partition, file_name) = match file_path.rsplit_once('/') {
+ Some((partition, name)) => (partition.to_string(), name.to_string()),
+ None => (String::new(), file_path.clone()),
+ };
+ let is_log_file = file_name.starts_with('.') && file_name.contains(".log.");
+ let request = requests_by_partition
+ .entry(partition.clone())
+ .or_insert_with(|| HoodieRollbackRequest {
+ partition_path: partition.clone(),
+ file_id: None,
+ latest_base_instant: None,
+ files_to_be_deleted: Vec::new(),
+ log_blocks_to_be_deleted: None,
Review Comment:
A crash after `rollback.requested` leaves a rollback that rs never resumes, and when Spark re-executes it the plan shape (log files in `logBlocksToBeDeleted`, per-partition requests, relative paths) trips v8 RollbackHelper. I pushed [1fce0dd1](https://github.com/apache/hudi-rs/pull/666/commits/1fce0dd1854be19dcb7ea67542d7a543222a1468), which writes Java's v8 marker-based plan shape and resumes pending rollbacks on their own instant, plus [955578c5](https://github.com/apache/hudi-rs/pull/666/commits/955578c57e02ea9b691abd0e95dd8316d33bb401), [4d04be08](https://github.com/apache/hudi-rs/pull/666/commits/4d04be08df9d662b1d8293b774ff76f3cb95e2c7), [006102cb](https://github.com/apache/hudi-rs/pull/666/commits/006102cbec9ad29fc44055f3649feca62bc5a8af) and [a1b86361](https://github.com/apache/hudi-rs/pull/666/commits/a1b863612631bbf8a7c945f1b185ac507353639f) for Java plan paths and scheme spellings, unreadable inflight plans, and refusing to resume a rollback whose target is still comple
ted.
##########
crates/core/src/write/rewrite.rs:
##########
@@ -0,0 +1,3924 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use arrow::array::{Array, ArrayRef, BooleanArray, StringArray, UInt32Array};
+use arrow::compute::concat_batches;
+use arrow::record_batch::RecordBatch;
+use arrow_select::take::take;
+use arrow_select::zip::zip;
+
+use crate::Result;
+use crate::config::table::HudiTableConfig::{OrderingFields, RecordKeyFields, RecordMergeStrategy};
+use crate::error::CoreError;
+use crate::expr::ExprOperator;
+use crate::expr::filter::{Filter, filters_to_row_mask, validate_fields_against_schemas};
+use crate::file_group::record_batches::RecordBatches;
+use crate::index::{HoodieIndex, HoodieKey, is_record_index_enabled};
+use crate::merge::RecordMergeStrategyValue;
+use crate::merge::record_merger::RecordMerger;
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::RecordIndexEntry;
+use crate::table::{ReadOptions, Table};
+use crate::timeline::instant::{Action, Instant};
+use crate::write::append::{
+ ensure_copy_on_write, generate_instant_time, is_layout_two, prepare_batches_for_write,
+ timeline_dir,
+};
+use crate::write::keygen::{hoodie_keys_for_batch, relative_data_path};
+use crate::write::metadata::{
+ instant_to_epoch_millis, is_column_stats_enabled, update_column_stats_partitions,
+ update_files_partition_entries, update_record_index,
+};
+
+/// Result of an upsert, delete, or overwrite write.
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub struct WriteResult {
+ /// Completed instant timestamp.
+ pub instant: String,
+ /// Number of rows written to replacement base files.
+ pub num_writes: usize,
+ /// Number of input keys that replaced existing rows.
+ pub num_updates: usize,
+ /// Number of input keys not previously present.
+ pub num_inserts: usize,
+ /// Number of rows deleted.
+ pub num_deletes: usize,
+}
+
+/// Options controlling an upsert.
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub struct UpsertOptions {
+ /// Data columns to overwrite for matched rows. `None` replaces complete rows.
+ pub update_columns: Option<Vec<String>>,
+}
+
+/// How a COW rewrite commits: same file-group slice vs replacecommit overwrite.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+enum RewriteKind {
+ /// New base file slice in the same file group(s); timeline action = commit.
+ Commit,
+ /// Replace file groups; timeline action = replacecommit (overwrite only for now).
+ Replace,
+}
+
+/// Critical section 1: request the action — mint the instant time and fence
+/// it (requested + inflight) under the table lock, then release; the write's
+/// heavy work runs unlocked. `request_commit` / `request_replacecommit` pick
+/// the timeline action from the rewrite kind.
+async fn request_rewrite_instant(
+ table: &Table,
+ operation: &str,
+ kind: RewriteKind,
+) -> Result<String> {
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ let _cs1 = lock.lock().await?;
+ let instant = generate_instant_time().await;
+ let action = match kind {
+ RewriteKind::Commit => Action::Commit,
+ RewriteKind::Replace => Action::ReplaceCommit,
+ };
+ let requested_bytes = if kind == RewriteKind::Replace {
+ crate::metadata::replace_commit::HoodieRequestedReplaceMetadata::for_operation(operation)
+ .to_avro_bytes()?
+ } else {
+ Vec::new()
+ };
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir(table),
+ &instant,
+ action,
+ requested_bytes,
+ crate::write::inflight_commit_metadata_bytes(operation, is_layout_two(table))?,
+ )
+ .await?;
+ Ok(instant)
+}
+
+/// Critical section 1 for MOR deltacommits.
+async fn request_deltacommit(table: &Table, operation: &str) -> Result<String> {
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ let _cs1 = lock.lock().await?;
+ let instant = generate_instant_time().await;
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir(table),
+ &instant,
+ Action::DeltaCommit,
+ Vec::new(),
+ crate::write::inflight_commit_metadata_bytes(operation, is_layout_two(table))?,
+ )
+ .await?;
+ Ok(instant)
+}
+
+pub async fn upsert_batches(
+ table: &mut Table,
+ batches: &[RecordBatch],
+ options: UpsertOptions,
+) -> Result<WriteResult> {
+ table.reload_timeline_for_write().await?;
+ ensure_configured_record_key(table)?;
+ if table.is_mor() {
+ return mor_upsert_batches(table, batches, options).await;
+ }
+ ensure_rewrite_supported(table)?;
+ if batches.is_empty() {
+ return Err(CoreError::Write(
+ "upsert requires at least one RecordBatch".to_string(),
+ ));
+ }
+ validate_write_input(table, batches, "upsert").await?;
+ if let Some(columns) = &options.update_columns {
+ let schema = batches[0].schema();
+ for column in columns {
+ if schema.field_with_name(column).is_err() {
+ return Err(CoreError::Schema(format!(
+ "update column '{column}' is not in the write batch schema"
+ )));
+ }
+ if MetaField::field_names_with_operation().contains(&column.as_str()) {
+ return Err(CoreError::Write(format!(
+ "update column '{column}' is a Hudi meta field"
+ )));
+ }
+ }
+ }
+ crate::write::keygen::validate_keygen_inputs(&table.hudi_configs, batches)?;
+ let instant = request_rewrite_instant(table, "UPSERT", RewriteKind::Commit).await?;
+ let incoming = prepare_batches_for_write(&table.hudi_configs, batches, &instant, "pending")?;
+ let incoming = concat(&incoming)?;
+ let key_name = record_key_name(table, &incoming)?;
+ let incoming = deduplicate_last_by_key(&incoming, &key_name, &table.hudi_configs)?;
+ let incoming_hoodie_keys = hoodie_keys_for_batch(&table.hudi_configs, &incoming, None)?;
+ let locations = crate::index::for_table_checked(table)
+ .await
+ .tag_location(table, &incoming_hoodie_keys)
+ .await?;
+ // Small-file packing (Java UpsertPartitioner.assignInserts): assign insert
+ // rows to existing small file groups first; chosen groups join the read
+ // set so their rows are preserved through the rewrite.
+ let sizing = crate::write::sizing::SizingConfig::from_table(table);
+ let avg_record_size = crate::write::sizing::average_record_size(
+ table,
+ table.file_system_view.storage.as_ref(),
+ &sizing,
+ )
+ .await;
+ let slices_by_partition = crate::write::sizing::latest_slices_by_partition(table).await?;
+ let mut insert_rows_by_partition: HashMap<String, Vec<usize>> = HashMap::new();
+ for (index, hoodie_key) in incoming_hoodie_keys.iter().enumerate() {
+ if locations.get(hoodie_key).and_then(Option::as_ref).is_none() {
+ insert_rows_by_partition
+ .entry(hoodie_key.partition_path.clone())
+ .or_default()
+ .push(index);
+ }
+ }
+ let mut insert_target_by_row: HashMap<usize, Option<String>> = HashMap::new();
+ for (partition, rows) in &insert_rows_by_partition {
+ let slices: Vec<&crate::file_group::file_slice::FileSlice> = slices_by_partition
+ .get(partition)
+ .map(|v| v.iter().collect())
+ .unwrap_or_default();
+ let small = crate::write::sizing::small_file_groups(&slices, false, &sizing);
+ let assignments = crate::write::sizing::assign_insert_buckets(
+ rows.len(),
+ &small,
+ avg_record_size,
+ &sizing,
+ );
+ let mut cursor = 0usize;
+ for (bucket, count) in assignments {
+ for row in &rows[cursor..cursor + count] {
+ match &bucket {
+ crate::write::sizing::InsertBucket::Existing { file_id } => {
+ insert_target_by_row.insert(*row, Some(file_id.clone()));
+ }
+ crate::write::sizing::InsertBucket::New => {
+ insert_target_by_row.insert(*row, None);
+ }
+ }
+ }
+ cursor += count;
+ }
+ }
+ // MergeHandle model: route each input row to its target file group, then
+ // per-group workers read+merge+write. The driver only holds the input and
+ // plan metadata.
+ let mut slice_by_file_id: HashMap<String, (String, crate::file_group::file_slice::FileSlice)> =
+ HashMap::new();
+ for (partition, slices) in &slices_by_partition {
+ for slice in slices {
+ slice_by_file_id.insert(
+ slice.file_id().to_string(),
+ (partition.clone(), slice.clone()),
+ );
+ }
+ }
+
+ let mut updates = 0usize;
+ let mut inserts = 0usize;
+ let mut rows_by_group: HashMap<String, Vec<u32>> = HashMap::new();
+ let mut new_rows_by_partition: HashMap<String, Vec<u32>> = HashMap::new();
+ let mut row_target: Vec<Option<String>> = Vec::with_capacity(incoming.num_rows());
+ // Update-partition-path (global index): a key upserted with a different
+ // partition value MOVES — evicted from the group in its old partition and
+ // inserted where its values say. Anything else leaves rows whose stored
+ // partition columns disagree with the path they physically live in.
+ let mut evictions_by_group: HashMap<String, HashSet<String>> = HashMap::new();
+ // Merge-before-move (Java parity): find the cross-partition updates whose
+ // incoming row loses the event-time merge; they stay in their old group.
+ let mut moved_candidates: HashMap<String, Vec<String>> = HashMap::new();
Review Comment:
Is always moving the record on a partition change intentional when RLI is the index? Java's `hoodie.record.index.update.partition.path` defaults to false and updates the record in its old partition, so the same upserts can leave a different layout than Spark would; I think keeping the move is fine, and I pushed [cf7975a5](https://github.com/apache/hudi-rs/pull/666/commits/cf7975a5039e91c5a01f1e39d72cc53527220547) to document the behavior in the design doc.
##########
crates/core/src/write/metadata.rs:
##########
@@ -0,0 +1,1320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Metadata table bootstrap and files / record_index / stats partition updates.
+
+use std::collections::{BTreeMap, HashMap};
+
+use crate::Result;
+use crate::file_group::log_file::scanner::{LogFileScanner, ScanResult};
+use crate::file_group::log_file::writer::LogFileWriter;
+use crate::file_group::log_file::{BlockMetadataKey, BlockType};
+use crate::hfile::{HFileReader, HFileWriter};
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::table::column_stats::{
+ ColumnRangeStats, aggregate_partition_stats, column_stats_for_file, column_stats_tombstones,
+ columns_to_index,
+};
+use crate::metadata::table::encode::{
+ ColumnStatsMetadata, FilesMetadataEntry, RecordIndexEntry, StatsIndexVersion,
+ decode_column_stats_entry, encode_all_partitions, encode_column_stats, encode_files_record,
+ encode_partition_stats, encode_record_index_entry, files_metadata_avro_schema_json,
+ hoodie_metadata_schema_json, record_index_metadata_avro_schema_json,
+};
+use crate::metadata::table::hash::{column_stats_index_key, partition_stats_index_key};
+use crate::metadata::table::records::{
+ FilesPartitionRecord, MetadataPartitionType, MetadataRecordType,
+};
+use crate::storage::Storage;
+use crate::table::{ReadOptions, Table};
+use crate::timeline::instant::{Action, Instant};
+use crate::timeline::selector::InstantRange;
+use arrow_schema::Schema;
+
+const METADATA_BASE: &str = ".hoodie/metadata";
+const FILES_FILE_ID: &str = "files-0000-0";
+/// Java `hoodie.metadata.record.index.min.filegroup.count` default.
+pub(crate) const DEFAULT_RLI_NUM_FILE_GROUPS: usize = 10;
+/// Java `hoodie.metadata.index.column.stats.file.group.count` default.
+pub(crate) const DEFAULT_COLUMN_STATS_NUM_FILE_GROUPS: usize = 2;
+/// Java `hoodie.metadata.index.partition.stats.file.group.count` default.
+pub(crate) const DEFAULT_PARTITION_STATS_NUM_FILE_GROUPS: usize = 1;
+
+fn column_stats_file_id(shard: usize) -> String {
+ format!("col-stats-{shard:04}-0")
+}
+
+fn partition_stats_file_id(shard: usize) -> String {
+ format!("partition-stats-{shard:04}-0")
+}
+
+/// A parquet (or deleted) data file whose column ranges should update MDT stats.
+#[derive(Debug, Clone)]
+pub struct StatsFileUpdate {
+ pub partition_path: String,
+ pub file_name: String,
+ pub is_deleted: bool,
+ /// Footer-derived ranges for adds; ignored when `is_deleted`.
+ pub ranges: Vec<ColumnRangeStats>,
+}
+
+fn record_index_file_id(shard: usize) -> String {
+ format!("record-index-{shard:04}-0")
+}
+
+/// Java `HoodieTableMetadataUtil.mapRecordKeyToFileGroupIndex` (31-hash, signed i32).
+pub(crate) fn map_record_key_to_file_group_index(
+ record_key: &str,
+ num_file_groups: usize,
+) -> usize {
+ if num_file_groups <= 1 {
+ return 0;
+ }
+ let mut h: i32 = 0;
+ // Java's String.hashCode iterates UTF-16 code units (charAt), not Unicode
+ // scalars; supplementary-plane keys must hash to the same shard as Java.
+ for unit in record_key.encode_utf16() {
+ h = h.wrapping_mul(31).wrapping_add(unit as i32);
+ }
+ (h.unsigned_abs() as usize) % num_file_groups
+}
+
+/// Java `HoodieBackedTableMetadataWriter.generateUniqueInstantTime` on a fresh table:
+/// `SOLO_COMMIT_TIMESTAMP` plus `offset` millis, zero-padded to 17 digits.
+fn bootstrap_instant(offset: usize) -> String {
+ format!("{offset:017}")
+}
+
+/// Create the metadata table properties and its initial partitions.
+///
+/// Each MDT partition is initialized in its own deltacommit with a unique
+/// instant (`00000000000000000`, `...001`, ...) in Java's enum order:
+/// files, column_stats, record_index, partition_stats.
+pub async fn bootstrap_metadata_table(
+ data_table_storage: &Storage,
+ table_name: &str,
+ table_version: isize,
+ record_index_enabled: bool,
+ column_stats_enabled: bool,
+ partition_stats_enabled: bool,
+) -> Result<()> {
+ let mdt_name = format!("{table_name}_metadata");
+ let checksum = crc32(format!(".{mdt_name}").as_bytes());
+ // The MDT's table version follows the data table's (Java initializes the
+ // MDT meta client with the data write config's version).
+ let properties = format!(
+ "# Generated by hudi-rs\n\
+ hoodie.table.name={mdt_name}\n\
+ hoodie.table.type=MERGE_ON_READ\n\
+ hoodie.table.base.file.format=HFILE\n\
+ hoodie.table.recordkey.fields=key\n\
+ hoodie.populate.meta.fields=false\n\
+ hoodie.table.version={table_version}\n\
+ hoodie.table.initial.version={table_version}\n\
+ hoodie.timeline.layout.version=2\n\
+ hoodie.timeline.path=timeline\n\
+ hoodie.timeline.history.path=history\n\
+ hoodie.archivelog.folder=history\n\
+ hoodie.table.keygenerator.type=HOODIE_TABLE_METADATA\n\
+ hoodie.compaction.payload.class=org.apache.hudi.metadata.HoodieMetadataPayload\n\
+ hoodie.record.merge.mode=CUSTOM\n\
+ hoodie.record.merge.strategy.id=00000000-0000-0000-0000-000000000000\n\
+ hoodie.datasource.write.drop.partition.columns=false\n\
+ hoodie.table.checksum={checksum}\n"
+ );
+ data_table_storage
+ .put_file(
+ &format!("{METADATA_BASE}/.hoodie/hoodie.properties"),
+ properties.into_bytes(),
+ )
+ .await?;
+ data_table_storage
+ .put_file(
+ &format!("{METADATA_BASE}/.hoodie/timeline/.keep"),
+ b"".as_slice(),
+ )
+ .await?;
+
+ // Java initializeFilesPartition on an empty table writes only the
+ // __all_partitions__ record with an empty partition list — no "." entry.
+ let files_instant = bootstrap_instant(0);
+ let all_partitions = encode_all_partitions(Vec::new())?;
+ let files_base_name = format!("{FILES_FILE_ID}_0-0-0_{files_instant}.hfile");
+ let hfile = HFileWriter::write(
+ &[(
+ FilesPartitionRecord::ALL_PARTITIONS_KEY.to_string(),
+ all_partitions,
+ )],
+ BTreeMap::from([(
+ "schema".to_string(),
+ files_metadata_avro_schema_json()?.as_bytes().to_vec(),
+ )]),
+ )
+ .map_err(|error| crate::error::CoreError::HFile(error.to_string()))?;
+ let files_base_size = hfile.len() as i64;
+ crate::write::ensure_partition_metadata(
+ data_table_storage,
+ &format!("{METADATA_BASE}/files"),
+ &files_instant,
+ )
+ .await?;
+ data_table_storage
+ .put_file(&format!("{METADATA_BASE}/files/{files_base_name}"), hfile)
+ .await?;
+ write_metadata_commit(
+ data_table_storage,
+ &files_instant,
+ HashMap::from([(
+ "files".to_string(),
+ vec![HoodieWriteStat {
+ file_id: Some(FILES_FILE_ID.to_string()),
+ path: Some(format!("files/{files_base_name}")),
+ base_file: Some(files_base_name),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(1),
+ total_write_bytes: Some(files_base_size),
+ file_size_in_bytes: Some(files_base_size),
+ partition_path: Some("files".to_string()),
+ ..Default::default()
+ }],
+ )]),
+ )
+ .await?;
+
+ // Remaining partitions in Java MetadataPartitionType enum order, each with
+ // the next unique bootstrap instant and its own deltacommit.
+ let mut next_offset = 1usize;
+ if column_stats_enabled {
+ bootstrap_empty_hfile_partition(
+ data_table_storage,
+ &bootstrap_instant(next_offset),
+ MetadataPartitionType::ColumnStats.partition_name(),
+ DEFAULT_COLUMN_STATS_NUM_FILE_GROUPS,
+ &column_stats_file_id,
+ hoodie_metadata_schema_json()?,
+ )
+ .await?;
+ next_offset += 1;
+ }
+ if record_index_enabled {
+ bootstrap_empty_hfile_partition(
+ data_table_storage,
+ &bootstrap_instant(next_offset),
+ MetadataPartitionType::RecordIndex.partition_name(),
+ DEFAULT_RLI_NUM_FILE_GROUPS,
+ &record_index_file_id,
+ record_index_metadata_avro_schema_json()?,
+ )
+ .await?;
+ next_offset += 1;
+ }
+ if partition_stats_enabled {
+ bootstrap_empty_hfile_partition(
+ data_table_storage,
+ &bootstrap_instant(next_offset),
+ MetadataPartitionType::PartitionStats.partition_name(),
+ DEFAULT_PARTITION_STATS_NUM_FILE_GROUPS,
+ &partition_stats_file_id,
+ hoodie_metadata_schema_json()?,
+ )
+ .await?;
+ }
+ Ok(())
+}
+
+async fn bootstrap_empty_hfile_partition(
+ storage: &Storage,
+ instant: &str,
+ partition_name: &str,
+ num_file_groups: usize,
+ file_id_fn: &dyn Fn(usize) -> String,
+ schema_json: &str,
+) -> Result<()> {
+ crate::write::ensure_partition_metadata(
+ storage,
+ &format!("{METADATA_BASE}/{partition_name}"),
+ instant,
+ )
+ .await?;
+ let mut stats = Vec::with_capacity(num_file_groups);
+ for shard in 0..num_file_groups {
+ let file_id = file_id_fn(shard);
+ let base_name = format!("{file_id}_0-0-0_{instant}.hfile");
+ let empty = HFileWriter::write(
+ &[],
+ BTreeMap::from([("schema".to_string(), schema_json.as_bytes().to_vec())]),
+ )
+ .map_err(|error| crate::error::CoreError::HFile(error.to_string()))?;
+ let size = empty.len() as i64;
+ storage
+ .put_file(
+ &format!("{METADATA_BASE}/{partition_name}/{base_name}"),
+ empty,
+ )
+ .await?;
+ stats.push(HoodieWriteStat {
+ file_id: Some(file_id),
+ path: Some(format!("{partition_name}/{base_name}")),
+ base_file: Some(base_name),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(0),
+ total_write_bytes: Some(size),
+ file_size_in_bytes: Some(size),
+ partition_path: Some(partition_name.to_string()),
+ ..Default::default()
+ });
+ }
+ write_metadata_commit(
+ storage,
+ instant,
+ HashMap::from([(partition_name.to_string(), stats)]),
+ )
+ .await
+}
+
+/// Append a files-partition HFile data block to the metadata table log file.
+///
+/// Returns the `files` partition write stats for the caller's single MDT
+/// deltacommit (see [`write_metadata_commit`]).
+pub async fn update_files_partition(
+ data_storage: &Storage,
+ instant: &str,
+ files: &[(String, String, i64)],
+) -> Result<Vec<HoodieWriteStat>> {
+ let entries = files
+ .iter()
+ .map(|(partition, file_name, size)| (partition.clone(), file_name.clone(), *size, false))
+ .collect::<Vec<_>>();
+ update_files_partition_entries(data_storage, instant, &entries).await
+}
+
+/// Apply file additions and deletions to the metadata table files partition.
+///
+/// Returns the `files` partition write stats for the caller's single MDT
+/// deltacommit (see [`write_metadata_commit`]).
+pub async fn update_files_partition_entries(
+ data_storage: &Storage,
+ instant: &str,
+ files: &[(String, String, i64, bool)],
+) -> Result<Vec<HoodieWriteStat>> {
+ let mut by_partition: BTreeMap<String, Vec<FilesMetadataEntry>> = BTreeMap::new();
+ for (partition, file_name, size, is_deleted) in files {
+ let partition = if partition.is_empty() {
+ FilesPartitionRecord::NON_PARTITIONED_NAME.to_string()
+ } else {
+ partition.clone()
+ };
+ by_partition
+ .entry(partition)
+ .or_default()
+ .push(FilesMetadataEntry {
+ name: file_name.clone(),
+ size: *size,
+ is_deleted: *is_deleted,
+ });
+ }
+ let partitions = by_partition.keys().cloned().collect::<Vec<_>>();
+ let all_partition_entries: Vec<FilesMetadataEntry> = partitions
+ .iter()
+ .map(|name| FilesMetadataEntry {
+ name: name.clone(),
+ size: 0,
+ is_deleted: false,
+ })
+ .collect();
+ let mut entries = vec![(
+ FilesPartitionRecord::ALL_PARTITIONS_KEY.to_string(),
+ encode_files_record(
+ FilesPartitionRecord::ALL_PARTITIONS_KEY,
+ MetadataRecordType::AllPartitions,
+ all_partition_entries,
+ )?,
+ )];
+ for (partition, files) in by_partition {
+ entries.push((
+ partition.clone(),
+ encode_files_record(&partition, MetadataRecordType::Files, files)?,
+ ));
+ }
+ let schema_json = files_metadata_avro_schema_json()?.to_string();
+ let hfile = HFileWriter::write(
+ &entries,
+ BTreeMap::from([("schema".to_string(), schema_json.as_bytes().to_vec())]),
+ )
+ .map_err(|error| crate::error::CoreError::HFile(error.to_string()))?;
+ // Java HFile data blocks embed SCHEMA (ordinal 2) alongside INSTANT_TIME.
+ let block = LogFileWriter::write_log_block(
+ BlockType::HfileData,
+ HashMap::from([
+ (BlockMetadataKey::InstantTime, instant.to_string()),
+ (BlockMetadataKey::Schema, schema_json),
+ ]),
+ &hfile,
+ );
+ let log_name = format!(".{FILES_FILE_ID}_{instant}.log.1_0-0-0");
+ let log_rel = format!("files/{log_name}");
+ let log_size = block.len() as i64;
+ data_storage
+ .put_file(&format!("{METADATA_BASE}/{log_rel}"), block)
+ .await?;
+ Ok(vec![HoodieWriteStat {
+ file_id: Some(FILES_FILE_ID.to_string()),
+ path: Some(log_rel),
+ log_files: Some(vec![log_name]),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(entries.len() as i64),
+ total_write_bytes: Some(log_size),
+ file_size_in_bytes: Some(log_size),
+ partition_path: Some("files".to_string()),
+ ..Default::default()
+ }])
+}
+
+/// Append record-index put/delete entries.
+///
+/// Puts are HFile data blocks. Deletes are v3 delete log blocks — never type-5
+/// records with null `recordIndexMetadata`, which NPEs Java's RLI reader.
+///
+/// Returns the `record_index` partition write stats for the caller's single
+/// MDT deltacommit (see [`write_metadata_commit`]).
+pub async fn update_record_index(
+ data_storage: &Storage,
+ instant: &str,
+ entries: &[RecordIndexEntry],
+) -> Result<Vec<HoodieWriteStat>> {
+ if entries.is_empty() {
+ return Ok(Vec::new());
+ }
+ let num_groups = detect_rli_num_file_groups(data_storage).await?;
+ let schema_json = record_index_metadata_avro_schema_json()?.to_string();
+ let mut puts_by_shard: HashMap<usize, Vec<(String, Vec<u8>)>> = HashMap::new();
+ let mut deletes_by_shard: HashMap<usize, Vec<(String, String)>> = HashMap::new();
+ for entry in entries {
+ let shard = map_record_key_to_file_group_index(&entry.record_key, num_groups);
+ if entry.is_deleted {
+ // DeleteRecord.partitionPath is the MDT partition (Java HoodieKey for RLI deletes).
+ deletes_by_shard.entry(shard).or_default().push((
+ entry.record_key.clone(),
+ MetadataPartitionType::RecordIndex
+ .partition_name()
+ .to_string(),
+ ));
+ } else {
+ let bytes = encode_record_index_entry(entry)?;
+ puts_by_shard
+ .entry(shard)
+ .or_default()
+ .push((entry.record_key.clone(), bytes));
+ }
+ }
+
+ let shards: std::collections::BTreeSet<usize> = puts_by_shard
+ .keys()
+ .chain(deletes_by_shard.keys())
+ .copied()
+ .collect();
+ let mut stats = Vec::new();
+ for shard in shards {
+ let file_id = record_index_file_id(shard);
+ let puts = puts_by_shard.remove(&shard).unwrap_or_default();
+ let deletes = deletes_by_shard.remove(&shard).unwrap_or_default();
+ let mut log_bytes = Vec::new();
+ let mut num_writes = 0i64;
+ if !puts.is_empty() {
+ num_writes += puts.len() as i64;
+ let hfile = HFileWriter::write(
+ &puts,
+ BTreeMap::from([("schema".to_string(), schema_json.as_bytes().to_vec())]),
+ )
+ .map_err(|error| crate::error::CoreError::HFile(error.to_string()))?;
+ log_bytes.extend(LogFileWriter::write_log_block(
+ BlockType::HfileData,
+ HashMap::from([
+ (BlockMetadataKey::InstantTime, instant.to_string()),
+ (BlockMetadataKey::Schema, schema_json.clone()),
+ ]),
+ &hfile,
+ ));
+ }
+ if !deletes.is_empty() {
+ num_writes += deletes.len() as i64;
+ log_bytes.extend(crate::write::build_delete_log_block(instant, &deletes, 0)?);
+ }
+ if log_bytes.is_empty() {
+ continue;
+ }
+ let log_name = format!(".{file_id}_{instant}.log.1_0-0-0");
+ let log_rel = format!("record_index/{log_name}");
+ let log_size = log_bytes.len() as i64;
+ data_storage
+ .put_file(&format!("{METADATA_BASE}/{log_rel}"), log_bytes)
+ .await?;
+ stats.push(HoodieWriteStat {
+ file_id: Some(file_id),
+ path: Some(log_rel),
+ log_files: Some(vec![log_name]),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(num_writes),
+ total_write_bytes: Some(log_size),
+ file_size_in_bytes: Some(log_size),
+ partition_path: Some("record_index".to_string()),
+ ..Default::default()
+ });
+ }
+ Ok(stats)
+}
+
+async fn detect_rli_num_file_groups(storage: &Storage) -> Result<usize> {
+ detect_num_file_groups(
+ storage,
+ MetadataPartitionType::RecordIndex.partition_name(),
+ "record-index-",
+ DEFAULT_RLI_NUM_FILE_GROUPS,
+ )
+ .await
+}
+
+async fn detect_num_file_groups(
Review Comment:
There's no guard for MDT layouts the writer doesn't maintain: a partitioned record index (Java 1.1+) misparses here, and bloom/secondary/expression index partitions go stale. I pushed [b46314dd](https://github.com/apache/hudi-rs/pull/666/commits/b46314dd961b76d847da65412cf0ce2336306e28), which refuses writes to tables whose metadata table has such partitions, in-flight partitions, or a partitioned record index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#issuecomment-6484952196
Any known limitations and improvements can be addressed in follow-ups. The initial hudi-rs writer support is strong enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4392345512
##########
.github/workflows/ci.yml:
##########
@@ -56,7 +56,10 @@ jobs:
os: [ ubuntu-24.04 ]
runs-on: ${{ matrix.os }}
container:
- image: xd009642/tarpaulin:0.32.7
+ # 0.32.7 produced broken coverage maps for async fn bodies in some
+ # build configurations (executed lines reported as uncovered, or absent
+ # from the map entirely), understating patch coverage.
Review Comment:
nit: remove verbose comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#issuecomment-5330867050
Coverage should now be fixed. How do we get the codecov report to regenerate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#issuecomment-6483047411
Pushed three commits on top of 8f7a8609 for the structure items in the follow-up list above:
- 1981756e **Complete every write through one shared sequence.** `write::complete_write` now runs the metadata table updates, lock, metadata table deltacommit, completed instant, bookkeeping and reload for append, MOR upsert, MOR delete and the COW rewrite finalizer (item 11), with `commit_metadata_bytes` building the instant's bytes once. One behavior change: the append commit's schema now leaves out the `_hoodie_*` meta fields like the other verbs and the Java writer; it recorded them when a read-back batch was appended.
- c75afef7 **Fold the table version 6 append path into `write::append`.** `table/writer.rs` is gone; `append_version_6` sits next to the version 8 path with what it leaves out stated on it, and `Table::append` routes through `append_batches` for every version (item 15). Its base file is written with the table's parquet writer properties like every other base file, and the refusal names `hoodie.table.record.merge.strategy`, the config that carries `append_only`.
- a4294584 **Split `write::rewrite` by concern** into `mod.rs` (public verbs), `validate.rs`, `mor.rs`, `rows.rs`, `cow.rs` and `lifecycle_tests.rs` (item 12). Code moves unchanged apart from `pub(super)` visibility.
Filed separately, since one writer per process is all that is supported for now: #857 for the process-wide timezone flag and the DST ordering of minted instants (item 3). Item 14 is covered by #856. Items 13 (`append.rs` as the shared utility module) and the remaining half of 15 stay on the list above; neither touches storage bytes or layout.
Verified with the full `hudi-core` suite (`--no-fail-fast`) and the `--no-default-features` clippy and test leg. The only failures are the two `gold_parity_tests` cases, which fail identically on 8f7a8609 on this macOS setup (chrono-tz formatter) and are unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#issuecomment-5323987435
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/666?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `73.13297%` with `885 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.70%. Comparing base ([`75b188e`](https://app.codecov.io/gh/apache/hudi-rs/commit/75b188ef2fbf7e3622c4424ff0d061c89be803b1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`111f6f9`](https://app.codecov.io/gh/apache/hudi-rs/commit/111f6f98a9536406d6547aaae4f11110e5a84ac7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/666?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [crates/core/tests/spark\_parity\_tests.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Ftests%2Fspark_parity_tests.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvdGVzdHMvc3BhcmtfcGFyaXR5X3Rlc3RzLnJz) | 1.79% | [164 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/write/create.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fwrite%2Fcreate.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3dyaXRlL2NyZWF0ZS5ycw==) | 45.31% | [146 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/metadata/table/encode.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fmetadata%2Ftable%2Fencode.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL21ldGFkYXRhL3RhYmxlL2VuY29kZS5ycw==) | 74.38% | [83 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/metadata/table/column\_stats.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fmetadata%2Ftable%2Fcolumn_stats.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL21ldGFkYXRhL3RhYmxlL2NvbHVtbl9zdGF0cy5ycw==) | 56.68% | [81 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/write/append.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fwrite%2Fappend.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3dyaXRlL2FwcGVuZC5ycw==) | 78.14% | [66 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/write/archival.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fwrite%2Farchival.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3dyaXRlL2FyY2hpdmFsLnJz) | 68.83% | [48 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/write/metadata.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fwrite%2Fmetadata.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3dyaXRlL21ldGFkYXRhLnJz) | 85.53% | [45 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/metadata/payload\_merger.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fmetadata%2Fpayload_merger.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL21ldGFkYXRhL3BheWxvYWRfbWVyZ2VyLnJz) | 55.38% | [29 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/write/keygen.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fwrite%2Fkeygen.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3dyaXRlL2tleWdlbi5ycw==) | 58.49% | [22 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/schema/avsc.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Fschema%2Favsc.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL3NjaGVtYS9hdnNjLnJz) | 68.25% | [20 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [26 more](https://app.codecov.io/gh/apache/hudi-rs/pull/666?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #666 +/- ##
==========================================
+ Coverage 80.30% 80.70% +0.40%
==========================================
Files 118 149 +31
Lines 11048 14908 +3860
==========================================
+ Hits 8872 12032 +3160
- Misses 2176 2876 +700
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hudi-rs/pull/666?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4406667406
##########
crates/core/src/config/table.rs:
##########
@@ -175,6 +181,7 @@ impl AsRef<str> for HudiTableConfig {
Self::TimelineHistoryPath => "hoodie.timeline.history.path",
Self::MetadataTableEnabled => "hoodie.metadata.enable",
Self::MetadataTablePartitions => "hoodie.table.metadata.partitions",
+ Self::RecordIndexEnabled => "hoodie.metadata.record.index.enable",
Review Comment:
This should use the new config key: `[hoodie.metadata.global.record.level.index.enable](https://hudi.apache.org/docs/configurations#hoodiemetadataglobalrecordlevelindexenable)` from the write config.
##########
crates/core/src/config/table.rs:
##########
@@ -145,6 +145,12 @@ pub enum HudiTableConfig {
/// When creating a metadata table instance, this value should be passed as the
/// PartitionFields option.
MetadataTablePartitions,
+
+ /// Enable the metadata-table record-level index (RLI) used to tag upsert/delete keys.
+ ///
+ /// When false, writers fall back to [`crate::index::SimpleIndex`] (full table scan).
+ /// Requires [`Self::MetadataTableEnabled`].
+ RecordIndexEnabled,
Review Comment:
Should this be inferred through `MetadataTablePartitions` instead of having an independent field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4395423067
##########
.github/workflows/ci.yml:
##########
@@ -56,7 +56,10 @@ jobs:
os: [ ubuntu-24.04 ]
runs-on: ${{ matrix.os }}
container:
- image: xd009642/tarpaulin:0.32.7
+ # 0.32.7 produced broken coverage maps for async fn bodies in some
+ # build configurations (executed lines reported as uncovered, or absent
+ # from the map entirely), understating patch coverage.
Review Comment:
Done in d2cdea39, the comment is gone.
##########
README.md:
##########
@@ -322,6 +322,107 @@ std::vector<std::string> log_file_paths{};
ArrowArrayStream* stream_ptr = reader->read_file_slice_from_paths("relative/path.parquet", log_file_paths);
```
+## Writing Tables (Rust)
Review Comment:
Done in a862d01c: `Usage Examples` is now `Reading Tables`, a sibling of `Writing Tables (Rust)`, and the release notes link points at the new anchor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4407284030
##########
crates/core/src/lib.rs:
##########
@@ -46,12 +46,17 @@
//! }
//! ```
+// The write-path async fns (e.g. `Table::delete_keys`) nest deeply enough that computing their
+// layout exceeds the default limit of 128 on Linux targets.
+#![recursion_limit = "256"]
+
Review Comment:
We should revisit why the nesting is deep.
##########
crates/core/src/metadata/commit.rs:
##########
@@ -193,21 +208,6 @@ impl HoodieCommitMetadata {
.unwrap_or_default()
}
- /// Get the file IDs to be replaced for a specific partition
- pub fn get_partition_replace_file_ids(&self, partition: &str) -> Option<&Vec<String>> {
Review Comment:
nit: should we keep the replaced file ID info and relevant tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#issuecomment-6489848116
Status of the follow-up list above, now that #666 has merged:
- Fixed in #851: 1 (9ee53f13, meta columns are stripped at every write entry point), 2 (45b2ceb5), 5 (d8d13c3b), 6 (5873c939), 7 (d174d68e), 8 (c7e8196d, a partial upsert under event-time ordering must set the ordering field), 9 (b976036d), 10 (da91977a), 12 (236f9044), 16 (d1479eb8), and 17 except the layout 1 rollback branch (5a4ca0cf). The duplicated `crc32` from 13 is gone too (ff6a0f9a).
- Tracked: 4 in #859, the rest of 13 in #861, the layout 1 rollback branch from 17 in #862, and `recursion_limit` in #860. 3 and 14 stay with #857 and #856.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4407455317
##########
crates/core/src/metadata/table/records.rs:
##########
@@ -282,7 +282,11 @@ pub fn decode_files_partition_record_with_schema(
});
}
- let avro_value = decode_avro_value(value, schema)?;
+ // Prefer the record's own writer schema (from the HFile or log block it
+ // was read from) — `schema` is only a fallback for records with no
+ // container schema. Mixed-writer tables (rs + Spark) have differing
+ // HoodieMetadataRecord schemas per container.
Review Comment:
The MDT writer schema mismatch between hudi-rs and Spark Java writers should already be fixed; let's revisit the logic and update the comment.
##########
crates/core/src/storage/mod.rs:
##########
@@ -398,6 +400,78 @@ impl Storage {
}
Ok(file_metadata)
}
+
+ /// Resolve a relative path to an [`ObjPath`] under this storage's base URL.
+ fn relative_obj_path(&self, relative_path: &str) -> Result<ObjPath> {
+ let obj_url = join_url_segments(&self.base_url, &[relative_path])?;
+ Ok(ObjPath::from_url_path(obj_url.path())?)
+ }
+
+ /// Write bytes to a path relative to the table base path.
+ pub async fn put_file(&self, relative_path: &str, bytes: impl Into<Bytes>) -> Result<()> {
+ let obj_path = self.relative_obj_path(relative_path)?;
+ self.object_store
+ .put(&obj_path, PutPayload::from(bytes.into()))
+ .await?;
+ Ok(())
+ }
+
+ /// Write a file only if nothing exists at the path (`PutMode::Create`).
+ ///
+ /// Timeline files must never silently replace one another: two writers
+ /// minting the same instant otherwise clobber each other's commit with no
+ /// error. An existing object surfaces as
+ /// [`StorageError::AlreadyExists`](error::StorageError) so callers can
+ /// report the conflict.
+ pub async fn put_file_if_absent(
Review Comment:
Should this use S3's conditional PUT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4404751515
##########
crates/core/src/write/append.rs:
##########
@@ -0,0 +1,1227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::collections::HashMap;
+use std::io::Cursor;
+use std::str::FromStr;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicI64, Ordering};
+
+use arrow::array::{ArrayRef, StringArray, UInt32Array};
+use arrow::record_batch::RecordBatch;
+use arrow_select::take::take;
+use chrono::Utc;
+use parquet::arrow::ArrowWriter;
+
+use crate::Result;
+use crate::config::table::HudiTableConfig::{
+ PopulatesMetaFields, RecordMergeStrategy, TableType, TableVersion, TimelineLayoutVersion,
+ TimelinePath,
+};
+use crate::config::table::TableTypeValue;
+use crate::error::CoreError;
+use crate::index::is_record_index_enabled;
+use crate::merge::RecordMergeStrategyValue;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::RecordIndexEntry;
+use crate::table::Table;
+use crate::timeline::instant::{Action, Instant};
+use crate::write::keygen::{
+ hoodie_keys_for_batch, hoodie_keys_for_batch_with_offset, relative_data_path,
+};
+use crate::write::metadata::{
+ is_column_stats_enabled, update_column_stats_partitions, update_files_partition,
+ update_record_index,
+};
+
+/// Result of an append write.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct AppendResult {
+ pub instant: String,
+ pub commit_relative_path: String,
+ pub base_file_path: String,
+ pub num_rows: usize,
+}
+
+static LAST_EPOCH_MILLIS: AtomicI64 = AtomicI64::new(0);
+
+/// Append one or more record batches as a single INSERT commit.
+pub async fn append_batches(table: &mut Table, batches: &[RecordBatch]) -> Result<AppendResult> {
+ append_batches_inner(table, batches, false).await
+}
+
+/// Append one or more record batches, requiring append-only merge mode.
+pub async fn append_batches_only(
+ table: &mut Table,
+ batches: &[RecordBatch],
+) -> Result<AppendResult> {
+ append_batches_inner(table, batches, true).await
+}
+
+async fn append_batches_inner(
+ table: &mut Table,
+ batches: &[RecordBatch],
+ require_append_only: bool,
+) -> Result<AppendResult> {
+ table.reload_timeline_for_write().await?;
+ if !table.is_mor() {
+ if require_append_only {
+ ensure_append_only_strategy(table)?;
+ } else {
+ ensure_insert_allowed(table)?;
+ }
+ }
+ ensure_supported_table_type(table)?;
+
+ if batches.is_empty() {
+ return Err(CoreError::Write(
+ "append requires at least one RecordBatch".to_string(),
+ ));
+ }
+ let schema = batches[0].schema();
+ for batch in batches.iter().skip(1) {
+ if batch.schema().as_ref() != schema.as_ref() {
+ return Err(CoreError::Write(
+ "All RecordBatches in append must share the same schema".to_string(),
+ ));
+ }
+ }
+ ensure_append_schema_matches_table(table, schema.as_ref()).await?;
+ // Must be expressible in commit metadata (Avro JSON) before fencing.
+ arrow_schema_to_avro_json(&strip_meta_fields_from_schema(schema.as_ref()))?;
+ let num_rows: usize = batches.iter().map(|b| b.num_rows()).sum();
+ if num_rows == 0 {
+ return Err(CoreError::Write(
+ "append requires at least one row".to_string(),
+ ));
+ }
+
+ crate::write::keygen::validate_keygen_inputs(&table.hudi_configs, batches)?;
+
+ let layout_two = is_layout_two(table);
+ let action = if table.is_mor() {
+ Action::DeltaCommit
+ } else {
+ Action::Commit
+ };
+ let timeline_dir = timeline_dir(table);
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ // Critical section 1: mint the instant and fence it, then work unlocked.
+ let cs1 = lock.lock().await?;
+ let request_instant = generate_instant_time().await;
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir,
+ &request_instant,
+ action.clone(),
+ Vec::new(),
+ crate::write::inflight_commit_metadata_bytes("INSERT", layout_two)?,
+ )
+ .await?;
+ drop(cs1);
+
+ // Group rows across batches by partition path and write one base file per partition.
+ let mut partition_rows: HashMap<String, Vec<(usize, usize)>> = HashMap::new();
+ for (batch_idx, batch) in batches.iter().enumerate() {
+ let keys = hoodie_keys_for_batch(&table.hudi_configs, batch, Some(&request_instant))?;
+ for (row_idx, key) in keys.iter().enumerate() {
+ partition_rows
+ .entry(key.partition_path.clone())
+ .or_default()
+ .push((batch_idx, row_idx));
+ }
+ }
+
+ let mut write_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ let mut files_mdt = Vec::new();
+ let mut rli_entries = Vec::new();
+ let mut stats_updates = Vec::new();
+ let mut written_paths: Vec<String> = Vec::new();
+ let mut primary_base_path = String::new();
+ let max_records = max_records_per_file(table);
+ // Monotonic auto-key row id across partitions and size-split chunks in this commit.
+ let mut auto_key_row_offset = 0usize;
+ let mut auto_key_partition_id = 0u32;
+
+ // Stable partition iteration so partition_id assignment is deterministic.
+ let mut partition_rows_sorted: Vec<_> = partition_rows.into_iter().collect();
+ partition_rows_sorted.sort_by(|a, b| a.0.cmp(&b.0));
+
+ // Plan every output file name up front and write markers before any data
+ // file is put — rollback of a crashed write is driven by these markers.
+ let mut file_plan: HashMap<String, std::collections::VecDeque<(String, String)>> =
+ HashMap::new();
+ let mut planned_markers = Vec::new();
+ for (partition_path, row_refs) in &partition_rows_sorted {
+ let chunks = row_refs.len().div_ceil(max_records).max(1);
+ let entries = file_plan.entry(partition_path.clone()).or_default();
+ for _ in 0..chunks {
+ let file_id = crate::write::new_file_id();
+ let file_name = format!("{file_id}_0-0-0_{request_instant}.parquet");
+ planned_markers.push(crate::write::markers::Marker::create(
+ partition_path,
+ &file_name,
+ ));
+ entries.push_back((file_id, file_name));
+ }
+ }
+ crate::write::markers::write_markers(storage.as_ref(), &request_instant, &planned_markers)
+ .await?;
+
+ // Phase 1 (sequential): slice + prepare every chunk, keeping table borrows
+ // out of the parallel phase.
+ struct ChunkPlan {
+ partition_path: String,
+ file_id: String,
+ file_name: String,
+ base_file_path: String,
+ row_count: i64,
+ chunk_keys: Vec<crate::index::HoodieKey>,
+ prepared: Vec<RecordBatch>,
+ }
+ let mut chunk_plans: Vec<ChunkPlan> = Vec::new();
+ for (partition_path, row_refs) in partition_rows_sorted {
+ // BTreeMap: deterministic row order in the written file.
+ let batch_indices: std::collections::BTreeMap<usize, Vec<u32>> = {
+ let mut map: std::collections::BTreeMap<usize, Vec<u32>> =
+ std::collections::BTreeMap::new();
+ for (batch_idx, row_idx) in &row_refs {
+ map.entry(*batch_idx).or_default().push(*row_idx as u32);
+ }
+ map
+ };
+ let mut partition_batches = Vec::new();
+ for (batch_idx, rows) in batch_indices {
+ partition_batches.push(take_rows(&batches[batch_idx], &rows)?);
+ }
+ let partition_batch = match partition_batches.pop() {
+ Some(only) if partition_batches.is_empty() => only,
+ Some(last) => {
+ partition_batches.push(last);
+ arrow::compute::concat_batches(&schema, &partition_batches)?
+ }
+ None => continue,
+ };
+ crate::write::ensure_partition_metadata(
+ storage.as_ref(),
+ &partition_path,
+ &request_instant,
+ )
+ .await?;
+
+ // Split oversized partitions into multiple base files (Java max.file.size).
+ let total_rows = partition_batch.num_rows();
+ let mut offset = 0usize;
+ while offset < total_rows {
+ let end = (offset + max_records).min(total_rows);
+ let indices: Vec<u32> = (offset as u32..end as u32).collect();
+ let chunk = take_rows(&partition_batch, &indices)?;
+ let chunk_row_offset = auto_key_row_offset;
+ let chunk_keys = hoodie_keys_for_batch_with_offset(
+ &table.hudi_configs,
+ &chunk,
+ Some(&request_instant),
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ auto_key_row_offset += chunk.num_rows();
+ let (file_id, file_name) = file_plan
+ .get_mut(&partition_path)
+ .and_then(std::collections::VecDeque::pop_front)
+ .ok_or_else(|| {
+ CoreError::Write(format!(
+ "file plan exhausted for partition '{partition_path}'"
+ ))
+ })?;
+ let base_file_path = relative_data_path(&partition_path, &file_name);
+ if primary_base_path.is_empty() {
+ primary_base_path = base_file_path.clone();
+ }
+ let prepared = prepare_batches_for_write_with_offset(
+ &table.hudi_configs,
+ &[chunk],
+ &request_instant,
+ &file_name,
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ chunk_plans.push(ChunkPlan {
+ partition_path: partition_path.clone(),
+ file_id,
+ file_name,
+ base_file_path,
+ row_count: indices.len() as i64,
+ chunk_keys,
+ prepared,
+ });
+ offset = end;
+ }
+ auto_key_partition_id = auto_key_partition_id.saturating_add(1);
+ }
+
+ // Phase 2 (parallel): encode + put each file on the write task pool.
+ let props = parquet_writer_props(table);
+ let collect_ranges = is_column_stats_enabled(table);
+ let tasks = chunk_plans
+ .iter()
+ .map(|plan| {
+ crate::write::parquet_file_task(
+ storage.clone(),
+ props.clone(),
+ plan.prepared.clone(),
+ plan.base_file_path.clone(),
+ collect_ranges,
+ )
+ })
+ .collect();
+ let results =
+ crate::write::run_write_tasks(tasks, crate::write::write_task_parallelism(table)).await;
+
+ // Phase 3 (sequential): assemble outputs in plan order; clean up on error.
+ let mut first_error = None;
+ for (plan, result) in chunk_plans.iter().zip(results) {
+ match result {
+ Ok(output) => {
+ written_paths.push(plan.base_file_path.clone());
+ if let Some(ranges) = output.ranges {
+ stats_updates.push(crate::write::metadata::StatsFileUpdate {
+ partition_path: plan.partition_path.clone(),
+ file_name: plan.file_name.clone(),
+ is_deleted: false,
+ ranges,
+ });
+ }
+ write_stats
+ .entry(plan.partition_path.clone())
+ .or_default()
+ .push(HoodieWriteStat {
+ file_id: Some(plan.file_id.clone()),
+ path: Some(plan.base_file_path.clone()),
+ // Basename only — FileGroup builder joins partition_path + name.
+ base_file: Some(plan.file_name.clone()),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(plan.row_count),
+ num_inserts: Some(plan.row_count),
+ total_write_bytes: Some(output.size),
+ file_size_in_bytes: Some(output.size),
+ partition_path: Some(plan.partition_path.clone()),
+ ..Default::default()
+ });
+ files_mdt.push((
+ plan.partition_path.clone(),
+ plan.file_name.clone(),
+ output.size,
+ ));
+ for key in &plan.chunk_keys {
+ rli_entries.push(RecordIndexEntry {
+ record_key: key.record_key.clone(),
+ partition_path: plan.partition_path.clone(),
+ file_id: plan.file_id.clone(),
+ instant_time_millis: crate::write::metadata::instant_to_epoch_millis_in(
+ &request_instant,
+ &crate::write::metadata::timeline_timezone(&table.hudi_configs),
+ ),
+ is_deleted: false,
+ });
+ }
+ }
+ Err(error) => {
+ if first_error.is_none() {
+ first_error = Some(error);
+ }
+ }
+ }
+ }
+ if let Some(error) = first_error {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+
+ let commit_metadata = HoodieCommitMetadata {
+ version: Some(1),
+ operation_type: Some("INSERT".to_string()),
+ partition_to_write_stats: Some(write_stats),
+ compacted: Some(false),
+ // Java HoodieCommitMetadata.extraMetadata is non-null; null → NPE in Spark.
+ extra_metadata: Some(HashMap::from([(
+ "schema".to_string(),
+ arrow_schema_to_avro_json(&schema)?,
+ )])),
+ };
+ let commit_bytes = if layout_two {
+ commit_metadata.to_avro_bytes()?
+ } else {
+ commit_metadata.to_json_bytes()?
+ };
+
+ // Crash-consistency order: data files → MDT files → MDT deltacommit →
+ // completed data commit. Finalizing the data timeline last means readers
+ // fence MDT contents by completed data instants. MDT log-file writes are
+ // "work" (unlocked); only completion runs in critical section 2.
+ let mut mdt_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ if table.is_metadata_table_enabled() {
+ let mdt_result = async {
+ mdt_stats.insert(
+ "files".to_string(),
+ update_files_partition(storage.as_ref(), &request_instant, &files_mdt).await?,
+ );
+ if is_record_index_enabled(table) {
+ let stats =
+ update_record_index(storage.as_ref(), &request_instant, &rli_entries).await?;
+ if !stats.is_empty() {
+ mdt_stats.insert("record_index".to_string(), stats);
+ }
+ }
+ if is_column_stats_enabled(table) && !stats_updates.is_empty() {
+ let stats = update_column_stats_partitions(
+ table,
+ &request_instant,
+ &stats_updates,
+ schema.as_ref(),
+ &[],
+ )
+ .await?;
+ mdt_stats.extend(stats);
+ }
+ Ok::<(), crate::error::CoreError>(())
+ }
+ .await;
+ if let Err(error) = mdt_result {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ }
+
+ // Critical section 2: complete the action (MDT deltacommit + data commit
+ // with a completion time minted under the lock), then bookkeeping.
+ let cs2 = lock.lock().await?;
+ if table.is_metadata_table_enabled()
+ && let Err(error) = crate::write::metadata::write_metadata_commit(
+ storage.as_ref(),
+ &request_instant,
+ mdt_stats,
+ )
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ let completion = if layout_two {
+ Some(generate_instant_time().await)
+ } else {
+ None
+ };
+ let instant = Instant::new_completed(request_instant.clone(), action, completion)?;
+ let commit_relative_path = instant.relative_path_with_base(&timeline_dir)?;
+ if let Err(error) = storage
Review Comment:
Fixed in 2992f536: data files are kept when the completed-commit PUT fails, since the PUT may have landed.
##########
crates/core/src/metadata/table/column_stats.rs:
##########
@@ -0,0 +1,638 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Extract column-range metadata from Parquet footers for MDT writes.
+
+use std::collections::HashMap;
+
+use arrow_schema::{DataType, Schema, TimeUnit};
+use bytes::Bytes;
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+use crate::Result;
+use crate::error::CoreError;
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::{ColumnStatValue, ColumnStatsMetadata};
+
+/// Per-column range extracted from a written Parquet base file.
+#[derive(Debug, Clone)]
+pub struct ColumnRangeStats {
+ pub column_name: String,
+ pub min_value: Option<ColumnStatValue>,
+ pub max_value: Option<ColumnStatValue>,
+ /// False when the parquet footer gave truncated/absent statistics, so the
+ /// bounds are UNKNOWN rather than absent. Readers treat a published record
+ /// with null min/max as "all nulls" and prune the file, so a record with
+ /// an unknown range must not be published at all (the file then falls into
+ /// the reader's not-indexed rescue set).
+ pub range_known: bool,
+ pub value_count: i64,
+ pub null_count: i64,
+ pub total_size: i64,
+ pub total_uncompressed_size: i64,
+}
+
+/// Read file-level column ranges from in-memory Parquet bytes (Java `ParquetUtils`).
+///
+/// Covers exactly [`columns_to_index`]: the three always-indexed meta fields plus
+/// the first 32 supported data fields. Nested (multi-part path) columns and types
+/// Spark does not index (bytes/fixed/decimal) are omitted rather than mis-encoded.
+pub fn column_ranges_from_parquet_bytes(bytes: &[u8]) -> Result<Vec<ColumnRangeStats>> {
+ let builder = ParquetRecordBatchReaderBuilder::try_new(Bytes::copy_from_slice(bytes))
+ .map_err(|e| CoreError::Write(format!("Failed to open Parquet for column stats: {e}")))?;
+ let schema = builder.schema().clone();
+ let metadata = builder.metadata().clone();
+ Ok(column_ranges_from_parquet_metadata(
+ metadata.as_ref(),
+ schema.as_ref(),
+ ))
+}
+
+fn column_ranges_from_parquet_metadata(
+ metadata: &parquet::file::metadata::ParquetMetaData,
+ schema: &Schema,
+) -> Vec<ColumnRangeStats> {
+ let type_by_name: HashMap<&str, &DataType> = schema
+ .fields()
+ .iter()
+ .map(|f| (f.name().as_str(), f.data_type()))
+ .collect();
+ let indexed_columns: std::collections::HashSet<String> =
+ columns_to_index(schema).into_iter().collect();
+
+ #[derive(Default)]
+ struct Acc {
+ min: Option<ColumnStatValue>,
+ max: Option<ColumnStatValue>,
+ value_count: i64,
+ null_count: i64,
+ total_size: i64,
+ total_uncompressed_size: i64,
+ data_type: Option<DataType>,
+ /// A chunk carried no statistics at all, so the accumulated range does
+ /// not cover the whole column and must not be published as a bound.
+ /// Truncated bounds do NOT set this: parquet truncates min downward
+ /// and max upward, so they stay valid (just not tight) bounds, and
+ /// Java publishes them as-is (ParquetUtils.readColumnStatsFromMetadata).
+ range_incomplete: bool,
+ }
+
+ let mut by_col: HashMap<String, Acc> = HashMap::new();
+ for row_group in metadata.row_groups() {
+ for col_chunk in row_group.columns() {
+ let path = col_chunk.column_descr().path();
+ if path.parts().len() != 1 {
+ continue;
+ }
+ let Some(col_name) = path.parts().first().map(|s| s.as_str()) else {
+ continue;
+ };
+ if !indexed_columns.contains(col_name) {
+ continue;
+ }
+ let Some(&data_type) = type_by_name.get(col_name) else {
+ continue;
+ };
+
+ let acc = by_col.entry(col_name.to_string()).or_default();
+ acc.data_type = Some(data_type.clone());
+ acc.value_count += col_chunk.num_values();
+ acc.total_size += col_chunk.compressed_size();
+ acc.total_uncompressed_size += col_chunk.uncompressed_size();
+
+ if let Some(stats) = col_chunk.statistics() {
+ // Null count only when actually recorded; never fabricate
+ // "all null" for chunks with stats disabled.
+ acc.null_count += stats.null_count_opt().unwrap_or(0) as i64;
+ match parquet_min_max_to_stat_values(stats, data_type) {
+ // Truncated bounds are still bounds; publish them (the
+ // record is never marked tight).
+ Some((min, max)) => {
+ acc.min = merge_min(acc.min.take(), Some(min));
+ acc.max = merge_max(acc.max.take(), Some(max));
+ }
+ // Parquet emits stubbed stats for an all-null chunk: no
+ // bounds exist, and none are needed — it cannot widen the
+ // column's range (Java equates nulls to the value count).
+ None if stats
+ .null_count_opt()
+ .is_some_and(|nulls| nulls as i64 == col_chunk.num_values()) => {}
+ None => acc.range_incomplete = true,
+ }
+ } else {
+ // Stats disabled for the chunk: range and null count unknown.
+ acc.range_incomplete = true;
+ }
+ }
+ }
+
+ let mut out: Vec<ColumnRangeStats> = by_col
+ .into_iter()
+ .map(|(column_name, acc)| ColumnRangeStats {
+ column_name,
+ range_known: !acc.range_incomplete,
+ min_value: (!acc.range_incomplete).then_some(acc.min).flatten(),
+ max_value: (!acc.range_incomplete).then_some(acc.max).flatten(),
+ value_count: acc.value_count,
+ null_count: acc.null_count,
+ total_size: acc.total_size,
+ total_uncompressed_size: acc.total_uncompressed_size,
+ })
+ .collect();
+ out.sort_by(|a, b| a.column_name.cmp(&b.column_name));
+ out
+}
+
+/// Build MDT column_stats payloads for a written (or deleted) file.
+pub fn column_stats_for_file(
+ file_name: &str,
+ ranges: &[ColumnRangeStats],
+ is_deleted: bool,
+) -> Vec<ColumnStatsMetadata> {
+ ranges
+ .iter()
+ .filter(|r| is_deleted || r.range_known || r.null_count == r.value_count)
+ .map(|r| ColumnStatsMetadata {
+ file_name: file_name.to_string(),
+ column_name: r.column_name.clone(),
+ min_value: if is_deleted {
+ None
+ } else {
+ r.min_value.clone()
+ },
+ max_value: if is_deleted {
+ None
+ } else {
+ r.max_value.clone()
+ },
+ value_count: if is_deleted { 0 } else { r.value_count },
+ null_count: if is_deleted { 0 } else { r.null_count },
+ total_size: if is_deleted { 0 } else { r.total_size },
+ total_uncompressed_size: if is_deleted {
+ 0
+ } else {
+ r.total_uncompressed_size
+ },
+ is_deleted,
+ is_tight_bound: false,
+ decoded_value_type_ordinal: None,
+ })
+ .collect()
+}
+
+/// Tombstone column_stats for a deleted file when footer ranges are unavailable.
+pub fn column_stats_tombstones(
+ file_name: &str,
+ column_names: &[String],
+) -> Vec<ColumnStatsMetadata> {
+ column_names
+ .iter()
+ .map(|column_name| ColumnStatsMetadata {
+ file_name: file_name.to_string(),
+ column_name: column_name.clone(),
+ min_value: None,
+ max_value: None,
+ value_count: 0,
+ null_count: 0,
+ total_size: 0,
+ total_uncompressed_size: 0,
+ is_deleted: true,
+ is_tight_bound: false,
+ decoded_value_type_ordinal: None,
+ })
+ .collect()
+}
+
+/// Aggregate file-level ranges into partition-level stats (same column name).
+pub fn aggregate_partition_stats(
+ partition_path: &str,
+ ranges_by_file: &[Vec<ColumnRangeStats>],
+ is_tight_bound: bool,
+) -> Vec<ColumnStatsMetadata> {
+ #[derive(Default)]
+ struct Acc {
+ min: Option<ColumnStatValue>,
+ max: Option<ColumnStatValue>,
+ value_count: i64,
+ null_count: i64,
+ total_size: i64,
+ total_uncompressed_size: i64,
+ range_incomplete: bool,
+ }
+ let mut by_col: HashMap<String, Acc> = HashMap::new();
+ for ranges in ranges_by_file {
+ for r in ranges {
+ let acc = by_col.entry(r.column_name.clone()).or_default();
+ // One file with unknown bounds makes the partition aggregate a
+ // non-tight (advisory) bound: publishing it tight would let
+ // pruning skip a partition whose unknown file may match.
+ if !r.range_known && r.null_count != r.value_count {
+ acc.range_incomplete = true;
+ }
+ acc.min = merge_min(acc.min.take(), r.min_value.clone());
+ acc.max = merge_max(acc.max.take(), r.max_value.clone());
+ acc.value_count += r.value_count;
+ acc.null_count += r.null_count;
+ acc.total_size += r.total_size;
+ acc.total_uncompressed_size += r.total_uncompressed_size;
+ }
+ }
+ // Java partition_stats stores the partition path in fileName.
+ let file_name = if partition_path.is_empty() {
+ ".".to_string()
+ } else {
+ partition_path.to_string()
+ };
+ let mut out: Vec<ColumnStatsMetadata> = by_col
+ .into_iter()
+ // A partition record with null bounds reads as "all nulls" and Spark's
+ // partition pruning has no not-indexed rescue set, so it would prune
+ // the whole partition. Publish nothing instead — unless the column
+ // really is all nulls, where null bounds are the correct encoding.
+ .filter(|(_, acc)| {
+ acc.min.is_some() || acc.max.is_some() || acc.null_count == acc.value_count
+ })
+ .map(|(column_name, acc)| ColumnStatsMetadata {
+ file_name: file_name.clone(),
+ column_name,
+ min_value: acc.min,
+ max_value: acc.max,
+ value_count: acc.value_count,
+ null_count: acc.null_count,
+ total_size: acc.total_size,
+ total_uncompressed_size: acc.total_uncompressed_size,
+ is_deleted: false,
+ is_tight_bound: is_tight_bound && !acc.range_incomplete,
+ decoded_value_type_ordinal: None,
+ })
+ .collect();
+ out.sort_by(|a, b| a.column_name.cmp(&b.column_name));
+ out
+}
+
+/// Java `HoodieMetadataConfig.COLUMN_STATS_INDEX_MAX_COLUMNS` default: only the
+/// first 32 supported data fields (schema order) are indexed.
+pub const MAX_COLUMNS_TO_INDEX: usize = 32;
+
+/// Columns eligible for MDT indexing from a table/data schema.
+///
+/// Java `HoodieTableMetadataUtil.getColumnsToIndex`: the three always-indexed
+/// meta fields (when present, i.e. `populateMetaFields`) plus the first
+/// [`MAX_COLUMNS_TO_INDEX`] supported non-meta fields in schema order.
+pub fn columns_to_index(schema: &Schema) -> Vec<String> {
+ // Java META_COLS_TO_ALWAYS_INDEX_SCHEMA_MAP is a TreeMap: alphabetical order.
+ let mut cols: Vec<String> = [
+ MetaField::CommitTime,
+ MetaField::PartitionPath,
+ MetaField::RecordKey,
+ ]
+ .iter()
+ .map(|m| m.as_ref().to_string())
+ .filter(|name| schema.fields().iter().any(|f| f.name() == name))
+ .collect();
+ cols.extend(
+ schema
+ .fields()
+ .iter()
+ .filter(|f| !MetaField::is_meta_field(f.name()) && is_indexable_type(f.data_type()))
+ .take(MAX_COLUMNS_TO_INDEX)
+ .map(|f| f.name().clone()),
+ );
+ cols
+}
+
+/// Java `isColumnTypeSupported` for the SPARK record type: nested types, enums,
+/// raw bytes/fixed — and hence decimals, stored as bytes/fixed — are not indexed.
+fn is_indexable_type(data_type: &DataType) -> bool {
+ matches!(
+ data_type,
+ DataType::Boolean
+ | DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::UInt8
+ | DataType::UInt16
+ | DataType::UInt32
+ | DataType::UInt64
+ | DataType::Float32
+ | DataType::Float64
+ | DataType::Utf8
+ | DataType::LargeUtf8
+ | DataType::Date32
+ | DataType::Time64(TimeUnit::Microsecond)
+ | DataType::Timestamp(_, _)
+ )
+}
+
+fn parquet_min_max_to_stat_values(
+ stats: &ParquetStatistics,
+ data_type: &DataType,
+) -> Option<(ColumnStatValue, ColumnStatValue)> {
+ match (stats, data_type) {
+ (ParquetStatistics::Boolean(s), DataType::Boolean) => Some((
+ ColumnStatValue::Boolean(*s.min_opt()?),
+ ColumnStatValue::Boolean(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Int32(s), DataType::Date32) => Some((
+ ColumnStatValue::Date(*s.min_opt()?),
+ ColumnStatValue::Date(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Int32(s), _) => Some((
+ ColumnStatValue::Int(*s.min_opt()?),
+ ColumnStatValue::Int(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Int64(s), DataType::Timestamp(unit, tz)) => {
+ let min = to_micros(*s.min_opt()?, *unit)?;
+ let max = to_micros(*s.max_opt()?, *unit)?;
+ // tz-less arrow timestamps are Java LOCAL_TIMESTAMP_* semantics.
+ if tz.is_some() {
+ Some((
+ ColumnStatValue::TimestampMicros(min),
+ ColumnStatValue::TimestampMicros(max),
+ ))
+ } else {
+ Some((
+ ColumnStatValue::LocalTimestampMicros(min),
+ ColumnStatValue::LocalTimestampMicros(max),
+ ))
+ }
+ }
+ (ParquetStatistics::Int64(s), DataType::Time64(TimeUnit::Microsecond)) => Some((
+ ColumnStatValue::TimeMicros(*s.min_opt()?),
+ ColumnStatValue::TimeMicros(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Int64(s), _) => Some((
+ ColumnStatValue::Long(*s.min_opt()?),
+ ColumnStatValue::Long(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Float(s), _) => Some((
+ ColumnStatValue::Float(*s.min_opt()?),
+ ColumnStatValue::Float(*s.max_opt()?),
+ )),
+ (ParquetStatistics::Double(s), _) => Some((
+ ColumnStatValue::Double(*s.min_opt()?),
+ ColumnStatValue::Double(*s.max_opt()?),
+ )),
+ (ParquetStatistics::ByteArray(s), DataType::Utf8 | DataType::LargeUtf8) => {
+ let min = String::from_utf8_lossy(s.min_opt()?.data()).into_owned();
+ let max = String::from_utf8_lossy(s.max_opt()?.data()).into_owned();
+ Some((ColumnStatValue::String(min), ColumnStatValue::String(max)))
+ }
+ (ParquetStatistics::ByteArray(s), _) => Some((
+ ColumnStatValue::Bytes(s.min_opt()?.data().to_vec()),
+ ColumnStatValue::Bytes(s.max_opt()?.data().to_vec()),
+ )),
+ (ParquetStatistics::FixedLenByteArray(s), _) => Some((
+ ColumnStatValue::Bytes(s.min_opt()?.data().to_vec()),
+ ColumnStatValue::Bytes(s.max_opt()?.data().to_vec()),
+ )),
+ _ => None,
+ }
+}
+
+fn to_micros(value: i64, unit: TimeUnit) -> Option<i64> {
+ match unit {
+ TimeUnit::Second => value.checked_mul(1_000_000),
+ TimeUnit::Millisecond => value.checked_mul(1_000),
+ TimeUnit::Microsecond => Some(value),
+ TimeUnit::Nanosecond => Some(value / 1_000),
+ }
+}
+
+fn merge_min(a: Option<ColumnStatValue>, b: Option<ColumnStatValue>) -> Option<ColumnStatValue> {
Review Comment:
Fixed in 7b08eb7f: partition_stats ranges merge in the column's own type.
##########
crates/core/src/write/append.rs:
##########
@@ -0,0 +1,1227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::collections::HashMap;
+use std::io::Cursor;
+use std::str::FromStr;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicI64, Ordering};
+
+use arrow::array::{ArrayRef, StringArray, UInt32Array};
+use arrow::record_batch::RecordBatch;
+use arrow_select::take::take;
+use chrono::Utc;
+use parquet::arrow::ArrowWriter;
+
+use crate::Result;
+use crate::config::table::HudiTableConfig::{
+ PopulatesMetaFields, RecordMergeStrategy, TableType, TableVersion, TimelineLayoutVersion,
+ TimelinePath,
+};
+use crate::config::table::TableTypeValue;
+use crate::error::CoreError;
+use crate::index::is_record_index_enabled;
+use crate::merge::RecordMergeStrategyValue;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::RecordIndexEntry;
+use crate::table::Table;
+use crate::timeline::instant::{Action, Instant};
+use crate::write::keygen::{
+ hoodie_keys_for_batch, hoodie_keys_for_batch_with_offset, relative_data_path,
+};
+use crate::write::metadata::{
+ is_column_stats_enabled, update_column_stats_partitions, update_files_partition,
+ update_record_index,
+};
+
+/// Result of an append write.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct AppendResult {
+ pub instant: String,
+ pub commit_relative_path: String,
+ pub base_file_path: String,
+ pub num_rows: usize,
+}
+
+static LAST_EPOCH_MILLIS: AtomicI64 = AtomicI64::new(0);
+
+/// Append one or more record batches as a single INSERT commit.
+pub async fn append_batches(table: &mut Table, batches: &[RecordBatch]) -> Result<AppendResult> {
+ append_batches_inner(table, batches, false).await
+}
+
+/// Append one or more record batches, requiring append-only merge mode.
+pub async fn append_batches_only(
+ table: &mut Table,
+ batches: &[RecordBatch],
+) -> Result<AppendResult> {
+ append_batches_inner(table, batches, true).await
+}
+
+async fn append_batches_inner(
+ table: &mut Table,
+ batches: &[RecordBatch],
+ require_append_only: bool,
+) -> Result<AppendResult> {
+ table.reload_timeline_for_write().await?;
+ if !table.is_mor() {
+ if require_append_only {
+ ensure_append_only_strategy(table)?;
+ } else {
+ ensure_insert_allowed(table)?;
+ }
+ }
+ ensure_supported_table_type(table)?;
+
+ if batches.is_empty() {
+ return Err(CoreError::Write(
+ "append requires at least one RecordBatch".to_string(),
+ ));
+ }
+ let schema = batches[0].schema();
+ for batch in batches.iter().skip(1) {
+ if batch.schema().as_ref() != schema.as_ref() {
+ return Err(CoreError::Write(
+ "All RecordBatches in append must share the same schema".to_string(),
+ ));
+ }
+ }
+ ensure_append_schema_matches_table(table, schema.as_ref()).await?;
+ // Must be expressible in commit metadata (Avro JSON) before fencing.
+ arrow_schema_to_avro_json(&strip_meta_fields_from_schema(schema.as_ref()))?;
+ let num_rows: usize = batches.iter().map(|b| b.num_rows()).sum();
+ if num_rows == 0 {
+ return Err(CoreError::Write(
+ "append requires at least one row".to_string(),
+ ));
+ }
+
+ crate::write::keygen::validate_keygen_inputs(&table.hudi_configs, batches)?;
+
+ let layout_two = is_layout_two(table);
+ let action = if table.is_mor() {
+ Action::DeltaCommit
+ } else {
+ Action::Commit
+ };
+ let timeline_dir = timeline_dir(table);
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ // Critical section 1: mint the instant and fence it, then work unlocked.
+ let cs1 = lock.lock().await?;
+ let request_instant = generate_instant_time().await;
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir,
+ &request_instant,
+ action.clone(),
+ Vec::new(),
+ crate::write::inflight_commit_metadata_bytes("INSERT", layout_two)?,
+ )
+ .await?;
+ drop(cs1);
+
+ // Group rows across batches by partition path and write one base file per partition.
+ let mut partition_rows: HashMap<String, Vec<(usize, usize)>> = HashMap::new();
+ for (batch_idx, batch) in batches.iter().enumerate() {
+ let keys = hoodie_keys_for_batch(&table.hudi_configs, batch, Some(&request_instant))?;
+ for (row_idx, key) in keys.iter().enumerate() {
+ partition_rows
+ .entry(key.partition_path.clone())
+ .or_default()
+ .push((batch_idx, row_idx));
+ }
+ }
+
+ let mut write_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ let mut files_mdt = Vec::new();
+ let mut rli_entries = Vec::new();
+ let mut stats_updates = Vec::new();
+ let mut written_paths: Vec<String> = Vec::new();
+ let mut primary_base_path = String::new();
+ let max_records = max_records_per_file(table);
+ // Monotonic auto-key row id across partitions and size-split chunks in this commit.
+ let mut auto_key_row_offset = 0usize;
+ let mut auto_key_partition_id = 0u32;
+
+ // Stable partition iteration so partition_id assignment is deterministic.
+ let mut partition_rows_sorted: Vec<_> = partition_rows.into_iter().collect();
+ partition_rows_sorted.sort_by(|a, b| a.0.cmp(&b.0));
+
+ // Plan every output file name up front and write markers before any data
+ // file is put — rollback of a crashed write is driven by these markers.
+ let mut file_plan: HashMap<String, std::collections::VecDeque<(String, String)>> =
+ HashMap::new();
+ let mut planned_markers = Vec::new();
+ for (partition_path, row_refs) in &partition_rows_sorted {
+ let chunks = row_refs.len().div_ceil(max_records).max(1);
+ let entries = file_plan.entry(partition_path.clone()).or_default();
+ for _ in 0..chunks {
+ let file_id = crate::write::new_file_id();
+ let file_name = format!("{file_id}_0-0-0_{request_instant}.parquet");
+ planned_markers.push(crate::write::markers::Marker::create(
+ partition_path,
+ &file_name,
+ ));
+ entries.push_back((file_id, file_name));
+ }
+ }
+ crate::write::markers::write_markers(storage.as_ref(), &request_instant, &planned_markers)
+ .await?;
+
+ // Phase 1 (sequential): slice + prepare every chunk, keeping table borrows
+ // out of the parallel phase.
+ struct ChunkPlan {
+ partition_path: String,
+ file_id: String,
+ file_name: String,
+ base_file_path: String,
+ row_count: i64,
+ chunk_keys: Vec<crate::index::HoodieKey>,
+ prepared: Vec<RecordBatch>,
+ }
+ let mut chunk_plans: Vec<ChunkPlan> = Vec::new();
+ for (partition_path, row_refs) in partition_rows_sorted {
+ // BTreeMap: deterministic row order in the written file.
+ let batch_indices: std::collections::BTreeMap<usize, Vec<u32>> = {
+ let mut map: std::collections::BTreeMap<usize, Vec<u32>> =
+ std::collections::BTreeMap::new();
+ for (batch_idx, row_idx) in &row_refs {
+ map.entry(*batch_idx).or_default().push(*row_idx as u32);
+ }
+ map
+ };
+ let mut partition_batches = Vec::new();
+ for (batch_idx, rows) in batch_indices {
+ partition_batches.push(take_rows(&batches[batch_idx], &rows)?);
+ }
+ let partition_batch = match partition_batches.pop() {
+ Some(only) if partition_batches.is_empty() => only,
+ Some(last) => {
+ partition_batches.push(last);
+ arrow::compute::concat_batches(&schema, &partition_batches)?
+ }
+ None => continue,
+ };
+ crate::write::ensure_partition_metadata(
+ storage.as_ref(),
+ &partition_path,
+ &request_instant,
+ )
+ .await?;
+
+ // Split oversized partitions into multiple base files (Java max.file.size).
+ let total_rows = partition_batch.num_rows();
+ let mut offset = 0usize;
+ while offset < total_rows {
+ let end = (offset + max_records).min(total_rows);
+ let indices: Vec<u32> = (offset as u32..end as u32).collect();
+ let chunk = take_rows(&partition_batch, &indices)?;
+ let chunk_row_offset = auto_key_row_offset;
+ let chunk_keys = hoodie_keys_for_batch_with_offset(
+ &table.hudi_configs,
+ &chunk,
+ Some(&request_instant),
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ auto_key_row_offset += chunk.num_rows();
+ let (file_id, file_name) = file_plan
+ .get_mut(&partition_path)
+ .and_then(std::collections::VecDeque::pop_front)
+ .ok_or_else(|| {
+ CoreError::Write(format!(
+ "file plan exhausted for partition '{partition_path}'"
+ ))
+ })?;
+ let base_file_path = relative_data_path(&partition_path, &file_name);
+ if primary_base_path.is_empty() {
+ primary_base_path = base_file_path.clone();
+ }
+ let prepared = prepare_batches_for_write_with_offset(
+ &table.hudi_configs,
+ &[chunk],
+ &request_instant,
+ &file_name,
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ chunk_plans.push(ChunkPlan {
+ partition_path: partition_path.clone(),
+ file_id,
+ file_name,
+ base_file_path,
+ row_count: indices.len() as i64,
+ chunk_keys,
+ prepared,
+ });
+ offset = end;
+ }
+ auto_key_partition_id = auto_key_partition_id.saturating_add(1);
+ }
+
+ // Phase 2 (parallel): encode + put each file on the write task pool.
+ let props = parquet_writer_props(table);
+ let collect_ranges = is_column_stats_enabled(table);
+ let tasks = chunk_plans
+ .iter()
+ .map(|plan| {
+ crate::write::parquet_file_task(
+ storage.clone(),
+ props.clone(),
+ plan.prepared.clone(),
+ plan.base_file_path.clone(),
+ collect_ranges,
+ )
+ })
+ .collect();
+ let results =
+ crate::write::run_write_tasks(tasks, crate::write::write_task_parallelism(table)).await;
+
+ // Phase 3 (sequential): assemble outputs in plan order; clean up on error.
+ let mut first_error = None;
+ for (plan, result) in chunk_plans.iter().zip(results) {
+ match result {
+ Ok(output) => {
+ written_paths.push(plan.base_file_path.clone());
+ if let Some(ranges) = output.ranges {
+ stats_updates.push(crate::write::metadata::StatsFileUpdate {
+ partition_path: plan.partition_path.clone(),
+ file_name: plan.file_name.clone(),
+ is_deleted: false,
+ ranges,
+ });
+ }
+ write_stats
+ .entry(plan.partition_path.clone())
+ .or_default()
+ .push(HoodieWriteStat {
+ file_id: Some(plan.file_id.clone()),
+ path: Some(plan.base_file_path.clone()),
+ // Basename only — FileGroup builder joins partition_path + name.
+ base_file: Some(plan.file_name.clone()),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(plan.row_count),
+ num_inserts: Some(plan.row_count),
+ total_write_bytes: Some(output.size),
+ file_size_in_bytes: Some(output.size),
+ partition_path: Some(plan.partition_path.clone()),
+ ..Default::default()
+ });
+ files_mdt.push((
+ plan.partition_path.clone(),
+ plan.file_name.clone(),
+ output.size,
+ ));
+ for key in &plan.chunk_keys {
+ rli_entries.push(RecordIndexEntry {
+ record_key: key.record_key.clone(),
+ partition_path: plan.partition_path.clone(),
+ file_id: plan.file_id.clone(),
+ instant_time_millis: crate::write::metadata::instant_to_epoch_millis_in(
+ &request_instant,
+ &crate::write::metadata::timeline_timezone(&table.hudi_configs),
+ ),
+ is_deleted: false,
+ });
+ }
+ }
+ Err(error) => {
+ if first_error.is_none() {
+ first_error = Some(error);
+ }
+ }
+ }
+ }
+ if let Some(error) = first_error {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+
+ let commit_metadata = HoodieCommitMetadata {
+ version: Some(1),
+ operation_type: Some("INSERT".to_string()),
+ partition_to_write_stats: Some(write_stats),
+ compacted: Some(false),
+ // Java HoodieCommitMetadata.extraMetadata is non-null; null → NPE in Spark.
+ extra_metadata: Some(HashMap::from([(
+ "schema".to_string(),
+ arrow_schema_to_avro_json(&schema)?,
+ )])),
+ };
+ let commit_bytes = if layout_two {
+ commit_metadata.to_avro_bytes()?
+ } else {
+ commit_metadata.to_json_bytes()?
+ };
+
+ // Crash-consistency order: data files → MDT files → MDT deltacommit →
+ // completed data commit. Finalizing the data timeline last means readers
+ // fence MDT contents by completed data instants. MDT log-file writes are
+ // "work" (unlocked); only completion runs in critical section 2.
+ let mut mdt_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ if table.is_metadata_table_enabled() {
+ let mdt_result = async {
+ mdt_stats.insert(
+ "files".to_string(),
+ update_files_partition(storage.as_ref(), &request_instant, &files_mdt).await?,
+ );
+ if is_record_index_enabled(table) {
+ let stats =
+ update_record_index(storage.as_ref(), &request_instant, &rli_entries).await?;
+ if !stats.is_empty() {
+ mdt_stats.insert("record_index".to_string(), stats);
+ }
+ }
+ if is_column_stats_enabled(table) && !stats_updates.is_empty() {
+ let stats = update_column_stats_partitions(
+ table,
+ &request_instant,
+ &stats_updates,
+ schema.as_ref(),
+ &[],
+ )
+ .await?;
+ mdt_stats.extend(stats);
+ }
+ Ok::<(), crate::error::CoreError>(())
+ }
+ .await;
+ if let Err(error) = mdt_result {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ }
+
+ // Critical section 2: complete the action (MDT deltacommit + data commit
+ // with a completion time minted under the lock), then bookkeeping.
+ let cs2 = lock.lock().await?;
+ if table.is_metadata_table_enabled()
+ && let Err(error) = crate::write::metadata::write_metadata_commit(
+ storage.as_ref(),
+ &request_instant,
+ mdt_stats,
+ )
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ let completion = if layout_two {
+ Some(generate_instant_time().await)
+ } else {
+ None
+ };
+ let instant = Instant::new_completed(request_instant.clone(), action, completion)?;
+ let commit_relative_path = instant.relative_path_with_base(&timeline_dir)?;
+ if let Err(error) = storage
+ .put_file_if_absent(&commit_relative_path, commit_bytes)
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error.into());
+ }
+ // The commit is durable from here: post-commit maintenance (markers,
+ // archival) and handle refresh are best-effort — a transient failure must
+ // not turn a committed write into an Err the caller would retry.
+ let _ =
+ crate::write::post_complete_bookkeeping(table, storage.as_ref(), &request_instant).await;
+ drop(cs2);
+
+ let _ = table.timeline.reload_completed_commits().await;
+ table.file_system_view.clear_cache();
+
+ Ok(AppendResult {
+ instant: request_instant,
+ commit_relative_path,
+ base_file_path: primary_base_path,
+ num_rows,
+ })
+}
+
+fn take_rows(batch: &RecordBatch, indices: &[u32]) -> Result<RecordBatch> {
+ let indices = UInt32Array::from(indices.to_vec());
+ let columns = batch
+ .columns()
+ .iter()
+ .map(|column| take(column.as_ref(), &indices, None))
+ .collect::<std::result::Result<Vec<_>, _>>()?;
+ RecordBatch::try_new(batch.schema(), columns).map_err(Into::into)
+}
+
+fn ensure_insert_allowed(table: &Table) -> Result<()> {
+ let strategy: String = table
+ .hudi_configs
+ .get_or_default(RecordMergeStrategy)
+ .into();
+ // Insert/append is valid for both append-only and upsertable (commit/event-time) tables.
+ match RecordMergeStrategyValue::from_str(&strategy)? {
+ RecordMergeStrategyValue::AppendOnly | RecordMergeStrategyValue::OverwriteWithLatest => {
+ Ok(())
+ }
+ }
+}
+
+fn ensure_append_only_strategy(table: &Table) -> Result<()> {
+ let strategy: String = table
+ .hudi_configs
+ .get_or_default(RecordMergeStrategy)
+ .into();
+ if RecordMergeStrategyValue::from_str(&strategy)? != RecordMergeStrategyValue::AppendOnly {
+ return Err(CoreError::Unsupported(
+ "append_only requires hoodie.record.merge.mode=append_only".to_string(),
+ ));
+ }
+ Ok(())
+}
+
+pub(crate) fn ensure_copy_on_write(table: &Table) -> Result<()> {
+ let table_type: String = table.hudi_configs.get(TableType)?.into();
+ if TableTypeValue::from_str(&table_type)? != TableTypeValue::CopyOnWrite {
+ return Err(CoreError::Unsupported(format!(
+ "append is currently only supported for COPY_ON_WRITE tables. Found '{table_type}'"
+ )));
+ }
+ Ok(())
+}
+
+fn ensure_supported_table_type(table: &Table) -> Result<()> {
+ let table_type: String = table.hudi_configs.get(TableType)?.into();
+ match TableTypeValue::from_str(&table_type)? {
+ TableTypeValue::CopyOnWrite | TableTypeValue::MergeOnRead => Ok(()),
+ }
+}
+
+pub(crate) fn is_layout_two(table: &Table) -> bool {
+ let table_version: isize = match table.hudi_configs.get(TableVersion) {
+ Ok(v) => v.into(),
+ Err(_) => return false,
+ };
+ let layout_version: isize = match table.hudi_configs.get(TimelineLayoutVersion) {
+ Ok(v) => v.into(),
+ Err(_) => return false,
+ };
+ table_version >= 8 && layout_version == 2
+}
+
+pub(crate) fn timeline_dir(table: &Table) -> String {
+ if is_layout_two(table) {
+ let timeline_path: String = table.hudi_configs.get_or_default(TimelinePath).into();
+ format!("{HUDI_METADATA_DIR}/{timeline_path}")
+ } else {
+ HUDI_METADATA_DIR.to_string()
+ }
+}
+
+/// Java `HoodieTimeGeneratorConfig` infers a 1 ms max expected clock skew when
+/// the (default) `InProcessLockProvider` is used — the single-writer case.
+const MAX_EXPECTED_CLOCK_SKEW_MS: u64 = 1;
+
+/// Skew-adjusting monotonic instant time generator (Java
+/// `SkewAdjustingTimeGenerator` + `HoodieInstantTimeGenerator.createNewInstantTime`).
+///
+/// Under a process-wide lock: capture the clock, wait out the max expected
+/// clock skew, and return the captured time — retrying (never bumping +1) until
+/// it exceeds every previously minted instant. Used for both requested and
+/// completion timestamps, so completion times are strictly greater than the
+/// requested times minted before them. Format `yyyyMMddHHmmssSSS` (UTC).
+/// Whether instants are formatted in local time (Java's
+/// `HoodieTimelineTimeZone.LOCAL`, the Hudi default) or UTC. Process-wide,
+/// mirroring `HoodieInstantTimeGenerator.setCommitTimeZone`; set from the
+/// table's `hoodie.table.timeline.timezone` when a table is created or
+/// loaded for writes. Spark writers mint LOCAL-time instants by default, so
+/// matching the table's declared zone keeps mixed-writer timelines ordered.
+static COMMIT_TIME_IS_LOCAL: std::sync::atomic::AtomicBool =
+ std::sync::atomic::AtomicBool::new(true);
+
+pub(crate) fn set_commit_timezone(timezone: &str) {
+ COMMIT_TIME_IS_LOCAL.store(
+ !timezone.eq_ignore_ascii_case("utc"),
+ std::sync::atomic::Ordering::Release,
+ );
+}
+
+pub(crate) async fn generate_instant_time() -> String {
+ static TIME_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
+ let _guard = TIME_LOCK.lock().await;
+ loop {
+ let now = Utc::now();
+ tokio::time::sleep(std::time::Duration::from_millis(MAX_EXPECTED_CLOCK_SKEW_MS)).await;
+ let candidate_millis = now.timestamp_millis();
+ // Store is race-free: all mints serialize through TIME_LOCK.
+ if candidate_millis > LAST_EPOCH_MILLIS.load(Ordering::Acquire) {
+ LAST_EPOCH_MILLIS.store(candidate_millis, Ordering::Release);
+ let format = "%Y%m%d%H%M%S%3f";
+ return if COMMIT_TIME_IS_LOCAL.load(std::sync::atomic::Ordering::Acquire) {
+ now.with_timezone(&chrono::Local).format(format).to_string()
+ } else {
+ now.format(format).to_string()
+ };
+ }
+ }
+}
+
+/// Default max base-file size (Java `hoodie.parquet.max.file.size` = 120 MiB).
+pub(crate) const DEFAULT_MAX_FILE_SIZE_BYTES: i64 = 120 * 1024 * 1024;
+/// Default record size estimate used for insert splits.
+pub(crate) const DEFAULT_RECORD_SIZE_ESTIMATE: i64 = 1024;
+
+pub(crate) fn max_records_per_file(table: &Table) -> usize {
+ let max_file_size = table
+ .hudi_configs
+ .as_options()
+ .get("hoodie.parquet.max.file.size")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_MAX_FILE_SIZE_BYTES)
+ .max(1);
+ let record_size = table
+ .hudi_configs
+ .as_options()
+ .get("hoodie.copyonwrite.record.size.estimate")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_RECORD_SIZE_ESTIMATE)
+ .max(1);
+ ((max_file_size / record_size) as usize).max(1)
+}
+
+/// Parquet `WriterProperties` resolved from `hoodie.parquet.*` configs.
+pub(crate) fn parquet_writer_props(table: &Table) -> parquet::file::properties::WriterProperties {
+ use parquet::basic::{Compression, GzipLevel, ZstdLevel};
+ use parquet::file::properties::WriterProperties;
+
+ let options = table.hudi_configs.as_options();
+ let page_size = options
+ .get("hoodie.parquet.page.size")
+ .and_then(|v| v.parse::<usize>().ok())
+ .unwrap_or(1024 * 1024)
+ .max(1);
+ let block_size = options
+ .get("hoodie.parquet.block.size")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_MAX_FILE_SIZE_BYTES)
+ .max(1);
+ let record_size = options
+ .get("hoodie.copyonwrite.record.size.estimate")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_RECORD_SIZE_ESTIMATE)
+ .max(1);
+ // parquet-rs row-group limit is row-count; approximate Java's byte block size.
+ let max_row_group_rows = ((block_size / record_size) as usize).max(1);
+ let dictionary_enabled = options
+ .get("hoodie.parquet.dictionary.enabled")
+ .map(|v| v.eq_ignore_ascii_case("true"))
+ .unwrap_or(true);
+ let codec = options
+ .get("hoodie.parquet.compression.codec")
+ .map(|s| s.as_str())
+ .unwrap_or("zstd");
+ let compression = match codec.to_ascii_lowercase().as_str() {
+ "uncompressed" | "none" => Compression::UNCOMPRESSED,
+ "snappy" => Compression::SNAPPY,
+ "gzip" | "gz" => Compression::GZIP(GzipLevel::try_new(6).unwrap_or_default()),
+ "lz4" => Compression::LZ4,
+ "brotli" => Compression::BROTLI(Default::default()),
+ // Default zstd — intentional vs Java's gzip; honor hoodie.parquet.compression.codec.
+ _ => Compression::ZSTD(ZstdLevel::try_new(3).unwrap_or_default()),
+ };
+
+ WriterProperties::builder()
+ .set_compression(compression)
+ .set_dictionary_enabled(dictionary_enabled)
+ .set_data_page_size_limit(page_size)
+ .set_dictionary_page_size_limit(page_size)
+ .set_max_row_group_row_count(Some(max_row_group_rows))
+ .build()
+}
+
+/// Encode batches to parquet bytes with pre-resolved properties (safe to run
+/// on a blocking thread with owned inputs).
+pub(crate) fn write_parquet_bytes_with_props(
+ props: parquet::file::properties::WriterProperties,
+ batches: &[RecordBatch],
+) -> Result<Vec<u8>> {
+ let cursor = Cursor::new(Vec::new());
+ let mut writer = ArrowWriter::try_new(cursor, batches[0].schema(), Some(props))?;
+ for batch in batches {
+ writer.write(batch)?;
+ }
+ let cursor = writer.into_inner()?;
+ Ok(cursor.into_inner())
+}
+
+/// Arrow→Avro schema JSON for `HoodieCommitMetadata.extraMetadata.schema`.
+///
+/// Unmapped Arrow types error out — silent `"string"` fallback corrupts Spark's
+/// TableSchemaResolver (Parquet types diverge from commit schema).
+pub(crate) fn arrow_schema_to_avro_json(schema: &arrow_schema::Schema) -> Result<String> {
+ let fields: Vec<String> = schema
+ .fields()
+ .iter()
+ .enumerate()
+ .map(|(index, f)| {
+ ensure_legal_avro_name(f.name())?;
+ // Field INDEX, not name: a name-derived path is not injective
+ // (`a.b` and `a_b` collide), and Avro refuses a redefined record.
+ let avro_type = arrow_type_to_avro_type_json_named(
+ f.data_type(),
+ &format!("hoodie_record_f{index}"),
+ )?;
+ let ty = if f.is_nullable() {
+ format!("[\"null\",{avro_type}]")
+ } else {
+ avro_type
+ };
+ Ok(format!(
+ "{{\"name\":\"{}\",\"type\":{}}}",
+ f.name().replace('\"', "\\\""),
+ ty
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+ Ok(format!(
+ "{{\"type\":\"record\",\"name\":\"hoodie_record\",\"fields\":[{}]}}",
+ fields.join(",")
+ ))
+}
+
+/// Avro field names must match [A-Za-z_][A-Za-z0-9_]*. A name that does not is
+/// rejected rather than renamed: renaming would diverge from the parquet column
+/// and silently mislabel the data, while an unparseable commit schema makes the
+/// table unreadable by every engine (including this one).
+fn ensure_legal_avro_name(name: &str) -> Result<()> {
+ let mut chars = name.chars();
+ let legal = chars
+ .next()
+ .is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
+ && chars.all(|c| c.is_ascii_alphanumeric() || c == '_');
+ if legal {
+ return Ok(());
+ }
+ Err(CoreError::Unsupported(format!(
+ "column name '{name}' is not a legal Avro name ([A-Za-z_][A-Za-z0-9_]*), \
+ so it cannot be written into Hudi commit metadata"
+ )))
+}
+
+/// `path` makes every nested Avro record name unique: Avro forbids redefining
+/// a named type, so a fixed name breaks any schema with two structs — Java's
+/// `Schema.Parser` rejects the whole commit schema.
+fn arrow_type_to_avro_type_json_named(dt: &arrow_schema::DataType, path: &str) -> Result<String> {
+ use arrow_schema::DataType;
+ match dt {
+ DataType::Boolean => Ok("\"boolean\"".to_string()),
+ DataType::Int8 | DataType::Int16 | DataType::Int32 => Ok("\"int\"".to_string()),
+ DataType::Int64 => Ok("\"long\"".to_string()),
+ DataType::Float32 => Ok("\"float\"".to_string()),
+ DataType::Float64 => Ok("\"double\"".to_string()),
+ DataType::Utf8 | DataType::LargeUtf8 => {
+ Ok("{\"type\":\"string\",\"avro.java.string\":\"String\"}".to_string())
+ }
+ DataType::Binary | DataType::LargeBinary => Ok("\"bytes\"".to_string()),
+ DataType::Date32 => Ok("{\"type\":\"int\",\"logicalType\":\"date\"}".to_string()),
+ DataType::Timestamp(unit, tz) => {
+ // A timezone-less Arrow timestamp is Spark's TimestampNTZType,
+ // which maps to Avro local-timestamp-*; mapping it to the
+ // UTC-adjusted logical type shifts values on read.
+ let local = tz.is_none();
+ let logical = match unit {
+ arrow_schema::TimeUnit::Millisecond if local => "local-timestamp-millis",
+ arrow_schema::TimeUnit::Microsecond if local => "local-timestamp-micros",
+ arrow_schema::TimeUnit::Millisecond => "timestamp-millis",
+ arrow_schema::TimeUnit::Microsecond => "timestamp-micros",
+ other => {
+ return Err(CoreError::Unsupported(format!(
+ "commit schema does not support Arrow timestamp unit {other:?}"
+ )));
+ }
+ };
+ Ok(format!(
+ "{{\"type\":\"long\",\"logicalType\":\"{logical}\"}}"
+ ))
+ }
+ DataType::Decimal128(precision, scale) | DataType::Decimal256(precision, scale) => {
+ Ok(format!(
+ "{{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":{precision},\"scale\":{scale}}}"
+ ))
+ }
+ DataType::List(field) | DataType::LargeList(field) => {
+ let item =
+ arrow_type_to_avro_type_json_named(field.data_type(), &format!("{path}_item"))?;
+ let item = if field.is_nullable() {
+ format!("[\"null\",{item}]")
+ } else {
+ item
+ };
+ Ok(format!("{{\"type\":\"array\",\"items\":{item}}}"))
+ }
+ DataType::Struct(fields) => {
+ let nested: Vec<String> = fields
+ .iter()
+ .enumerate()
+ .map(|(index, f)| {
+ ensure_legal_avro_name(f.name())?;
+ let avro_type = arrow_type_to_avro_type_json_named(
+ f.data_type(),
+ &format!("{path}f{index}"),
+ )?;
+ let ty = if f.is_nullable() {
+ format!("[\"null\",{avro_type}]")
+ } else {
+ avro_type
+ };
+ Ok(format!(
+ "{{\"name\":\"{}\",\"type\":{}}}",
+ f.name().replace('\"', "\\\""),
+ ty
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+ Ok(format!(
+ "{{\"type\":\"record\",\"name\":\"{path}_record\",\"fields\":[{}]}}",
+ nested.join(",")
+ ))
+ }
+ other => Err(CoreError::Unsupported(format!(
+ "cannot map Arrow type {other:?} to Avro for commit metadata schema"
+ ))),
+ }
+}
+
+async fn ensure_append_schema_matches_table(
+ table: &Table,
+ batch_schema: &arrow_schema::Schema,
+) -> Result<()> {
+ if table.timeline.completed_commits.is_empty() {
+ return Ok(());
+ }
+ let table_schema = table.get_schema().await?;
+ let batch_data = strip_meta_fields_from_schema(batch_schema);
+ if schemas_equal_by_fields(&table_schema, &batch_data) {
+ return Ok(());
+ }
+ Err(CoreError::Write(format!(
+ "append batch schema does not match table schema (table fields={:?}, batch fields={:?})",
+ table_schema
+ .fields()
+ .iter()
+ .map(|f| (f.name().clone(), f.data_type().clone()))
+ .collect::<Vec<_>>(),
+ batch_data
+ .fields()
+ .iter()
+ .map(|f| (f.name().clone(), f.data_type().clone()))
+ .collect::<Vec<_>>(),
+ )))
+}
+
+pub(crate) fn strip_meta_fields_from_schema(schema: &arrow_schema::Schema) -> arrow_schema::Schema {
+ let fields: Vec<_> = schema
+ .fields()
+ .iter()
+ .filter(|f| !f.name().starts_with("_hoodie_"))
+ .cloned()
+ .collect();
+ arrow_schema::Schema::new(fields)
+}
+
+/// Prepend the Hudi meta fields to a data schema when absent — written files
+/// always carry them, so index bookkeeping must consider them.
+pub(crate) fn schema_with_meta_fields(schema: &arrow_schema::Schema) -> arrow_schema::Schema {
+ // Match the meta fields by exact name: a data column may legitimately
+ // begin with `_hoodie_` (Hudi's own `_hoodie_is_deleted` marker does), and
+ // treating it as proof the meta fields are present would silently skip
+ // them.
+ let meta: std::collections::HashSet<&str> =
+ crate::metadata::meta_field::MetaField::field_names()
+ .into_iter()
+ .collect();
+ if schema
+ .fields()
+ .iter()
+ .any(|f| meta.contains(f.name().as_str()))
+ {
+ return schema.clone();
+ }
+ let mut fields: Vec<arrow_schema::FieldRef> =
+ crate::metadata::meta_field::MetaField::field_names()
+ .into_iter()
+ .map(|name| {
+ std::sync::Arc::new(arrow_schema::Field::new(
+ name,
+ arrow_schema::DataType::Utf8,
+ true,
+ ))
+ })
+ .collect();
+ fields.extend(schema.fields().iter().cloned());
+ arrow_schema::Schema::new(fields)
+}
+
+fn schemas_equal_by_fields(a: &arrow_schema::Schema, b: &arrow_schema::Schema) -> bool {
+ if a.fields().len() != b.fields().len() {
+ return false;
+ }
+ a.fields()
+ .iter()
+ .zip(b.fields().iter())
+ .all(|(fa, fb)| fa.name() == fb.name() && fa.data_type() == fb.data_type())
+}
+
+pub(crate) fn prepare_batches_for_write(
+ hudi_configs: &crate::config::HudiConfigs,
+ batches: &[RecordBatch],
+ instant: &str,
+ file_name: &str,
+) -> Result<Vec<RecordBatch>> {
+ prepare_batches_for_write_with_offset(hudi_configs, batches, instant, file_name, 0, 0)
+}
+
+pub(crate) fn prepare_batches_for_write_with_offset(
+ hudi_configs: &crate::config::HudiConfigs,
+ batches: &[RecordBatch],
+ instant: &str,
+ file_name: &str,
+ row_id_offset: usize,
+ partition_id: u32,
+) -> Result<Vec<RecordBatch>> {
+ let populates_meta_fields: bool = hudi_configs.get_or_default(PopulatesMetaFields).into();
+ if !populates_meta_fields {
+ return Ok(batches.to_vec());
+ }
+
+ batches
Review Comment:
Fixed in c01ec3c3: auto-generated keys are numbered across all batches of a write.
##########
crates/core/src/write/append.rs:
##########
@@ -0,0 +1,1227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::collections::HashMap;
+use std::io::Cursor;
+use std::str::FromStr;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicI64, Ordering};
+
+use arrow::array::{ArrayRef, StringArray, UInt32Array};
+use arrow::record_batch::RecordBatch;
+use arrow_select::take::take;
+use chrono::Utc;
+use parquet::arrow::ArrowWriter;
+
+use crate::Result;
+use crate::config::table::HudiTableConfig::{
+ PopulatesMetaFields, RecordMergeStrategy, TableType, TableVersion, TimelineLayoutVersion,
+ TimelinePath,
+};
+use crate::config::table::TableTypeValue;
+use crate::error::CoreError;
+use crate::index::is_record_index_enabled;
+use crate::merge::RecordMergeStrategyValue;
+use crate::metadata::HUDI_METADATA_DIR;
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::RecordIndexEntry;
+use crate::table::Table;
+use crate::timeline::instant::{Action, Instant};
+use crate::write::keygen::{
+ hoodie_keys_for_batch, hoodie_keys_for_batch_with_offset, relative_data_path,
+};
+use crate::write::metadata::{
+ is_column_stats_enabled, update_column_stats_partitions, update_files_partition,
+ update_record_index,
+};
+
+/// Result of an append write.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct AppendResult {
+ pub instant: String,
+ pub commit_relative_path: String,
+ pub base_file_path: String,
+ pub num_rows: usize,
+}
+
+static LAST_EPOCH_MILLIS: AtomicI64 = AtomicI64::new(0);
+
+/// Append one or more record batches as a single INSERT commit.
+pub async fn append_batches(table: &mut Table, batches: &[RecordBatch]) -> Result<AppendResult> {
+ append_batches_inner(table, batches, false).await
+}
+
+/// Append one or more record batches, requiring append-only merge mode.
+pub async fn append_batches_only(
+ table: &mut Table,
+ batches: &[RecordBatch],
+) -> Result<AppendResult> {
+ append_batches_inner(table, batches, true).await
+}
+
+async fn append_batches_inner(
+ table: &mut Table,
+ batches: &[RecordBatch],
+ require_append_only: bool,
+) -> Result<AppendResult> {
+ table.reload_timeline_for_write().await?;
+ if !table.is_mor() {
+ if require_append_only {
+ ensure_append_only_strategy(table)?;
+ } else {
+ ensure_insert_allowed(table)?;
+ }
+ }
+ ensure_supported_table_type(table)?;
+
+ if batches.is_empty() {
+ return Err(CoreError::Write(
+ "append requires at least one RecordBatch".to_string(),
+ ));
+ }
+ let schema = batches[0].schema();
+ for batch in batches.iter().skip(1) {
+ if batch.schema().as_ref() != schema.as_ref() {
+ return Err(CoreError::Write(
+ "All RecordBatches in append must share the same schema".to_string(),
+ ));
+ }
+ }
+ ensure_append_schema_matches_table(table, schema.as_ref()).await?;
+ // Must be expressible in commit metadata (Avro JSON) before fencing.
+ arrow_schema_to_avro_json(&strip_meta_fields_from_schema(schema.as_ref()))?;
+ let num_rows: usize = batches.iter().map(|b| b.num_rows()).sum();
+ if num_rows == 0 {
+ return Err(CoreError::Write(
+ "append requires at least one row".to_string(),
+ ));
+ }
+
+ crate::write::keygen::validate_keygen_inputs(&table.hudi_configs, batches)?;
+
+ let layout_two = is_layout_two(table);
+ let action = if table.is_mor() {
+ Action::DeltaCommit
+ } else {
+ Action::Commit
+ };
+ let timeline_dir = timeline_dir(table);
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ // Critical section 1: mint the instant and fence it, then work unlocked.
+ let cs1 = lock.lock().await?;
+ let request_instant = generate_instant_time().await;
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir,
+ &request_instant,
+ action.clone(),
+ Vec::new(),
+ crate::write::inflight_commit_metadata_bytes("INSERT", layout_two)?,
+ )
+ .await?;
+ drop(cs1);
+
+ // Group rows across batches by partition path and write one base file per partition.
+ let mut partition_rows: HashMap<String, Vec<(usize, usize)>> = HashMap::new();
+ for (batch_idx, batch) in batches.iter().enumerate() {
+ let keys = hoodie_keys_for_batch(&table.hudi_configs, batch, Some(&request_instant))?;
+ for (row_idx, key) in keys.iter().enumerate() {
+ partition_rows
+ .entry(key.partition_path.clone())
+ .or_default()
+ .push((batch_idx, row_idx));
+ }
+ }
+
+ let mut write_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ let mut files_mdt = Vec::new();
+ let mut rli_entries = Vec::new();
+ let mut stats_updates = Vec::new();
+ let mut written_paths: Vec<String> = Vec::new();
+ let mut primary_base_path = String::new();
+ let max_records = max_records_per_file(table);
+ // Monotonic auto-key row id across partitions and size-split chunks in this commit.
+ let mut auto_key_row_offset = 0usize;
+ let mut auto_key_partition_id = 0u32;
+
+ // Stable partition iteration so partition_id assignment is deterministic.
+ let mut partition_rows_sorted: Vec<_> = partition_rows.into_iter().collect();
+ partition_rows_sorted.sort_by(|a, b| a.0.cmp(&b.0));
+
+ // Plan every output file name up front and write markers before any data
+ // file is put — rollback of a crashed write is driven by these markers.
+ let mut file_plan: HashMap<String, std::collections::VecDeque<(String, String)>> =
+ HashMap::new();
+ let mut planned_markers = Vec::new();
+ for (partition_path, row_refs) in &partition_rows_sorted {
+ let chunks = row_refs.len().div_ceil(max_records).max(1);
+ let entries = file_plan.entry(partition_path.clone()).or_default();
+ for _ in 0..chunks {
+ let file_id = crate::write::new_file_id();
+ let file_name = format!("{file_id}_0-0-0_{request_instant}.parquet");
+ planned_markers.push(crate::write::markers::Marker::create(
+ partition_path,
+ &file_name,
+ ));
+ entries.push_back((file_id, file_name));
+ }
+ }
+ crate::write::markers::write_markers(storage.as_ref(), &request_instant, &planned_markers)
+ .await?;
+
+ // Phase 1 (sequential): slice + prepare every chunk, keeping table borrows
+ // out of the parallel phase.
+ struct ChunkPlan {
+ partition_path: String,
+ file_id: String,
+ file_name: String,
+ base_file_path: String,
+ row_count: i64,
+ chunk_keys: Vec<crate::index::HoodieKey>,
+ prepared: Vec<RecordBatch>,
+ }
+ let mut chunk_plans: Vec<ChunkPlan> = Vec::new();
+ for (partition_path, row_refs) in partition_rows_sorted {
+ // BTreeMap: deterministic row order in the written file.
+ let batch_indices: std::collections::BTreeMap<usize, Vec<u32>> = {
+ let mut map: std::collections::BTreeMap<usize, Vec<u32>> =
+ std::collections::BTreeMap::new();
+ for (batch_idx, row_idx) in &row_refs {
+ map.entry(*batch_idx).or_default().push(*row_idx as u32);
+ }
+ map
+ };
+ let mut partition_batches = Vec::new();
+ for (batch_idx, rows) in batch_indices {
+ partition_batches.push(take_rows(&batches[batch_idx], &rows)?);
+ }
+ let partition_batch = match partition_batches.pop() {
+ Some(only) if partition_batches.is_empty() => only,
+ Some(last) => {
+ partition_batches.push(last);
+ arrow::compute::concat_batches(&schema, &partition_batches)?
+ }
+ None => continue,
+ };
+ crate::write::ensure_partition_metadata(
+ storage.as_ref(),
+ &partition_path,
+ &request_instant,
+ )
+ .await?;
+
+ // Split oversized partitions into multiple base files (Java max.file.size).
+ let total_rows = partition_batch.num_rows();
+ let mut offset = 0usize;
+ while offset < total_rows {
+ let end = (offset + max_records).min(total_rows);
+ let indices: Vec<u32> = (offset as u32..end as u32).collect();
+ let chunk = take_rows(&partition_batch, &indices)?;
+ let chunk_row_offset = auto_key_row_offset;
+ let chunk_keys = hoodie_keys_for_batch_with_offset(
+ &table.hudi_configs,
+ &chunk,
+ Some(&request_instant),
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ auto_key_row_offset += chunk.num_rows();
+ let (file_id, file_name) = file_plan
+ .get_mut(&partition_path)
+ .and_then(std::collections::VecDeque::pop_front)
+ .ok_or_else(|| {
+ CoreError::Write(format!(
+ "file plan exhausted for partition '{partition_path}'"
+ ))
+ })?;
+ let base_file_path = relative_data_path(&partition_path, &file_name);
+ if primary_base_path.is_empty() {
+ primary_base_path = base_file_path.clone();
+ }
+ let prepared = prepare_batches_for_write_with_offset(
+ &table.hudi_configs,
+ &[chunk],
+ &request_instant,
+ &file_name,
+ chunk_row_offset,
+ auto_key_partition_id,
+ )?;
+ chunk_plans.push(ChunkPlan {
+ partition_path: partition_path.clone(),
+ file_id,
+ file_name,
+ base_file_path,
+ row_count: indices.len() as i64,
+ chunk_keys,
+ prepared,
+ });
+ offset = end;
+ }
+ auto_key_partition_id = auto_key_partition_id.saturating_add(1);
+ }
+
+ // Phase 2 (parallel): encode + put each file on the write task pool.
+ let props = parquet_writer_props(table);
+ let collect_ranges = is_column_stats_enabled(table);
+ let tasks = chunk_plans
+ .iter()
+ .map(|plan| {
+ crate::write::parquet_file_task(
+ storage.clone(),
+ props.clone(),
+ plan.prepared.clone(),
+ plan.base_file_path.clone(),
+ collect_ranges,
+ )
+ })
+ .collect();
+ let results =
+ crate::write::run_write_tasks(tasks, crate::write::write_task_parallelism(table)).await;
+
+ // Phase 3 (sequential): assemble outputs in plan order; clean up on error.
+ let mut first_error = None;
+ for (plan, result) in chunk_plans.iter().zip(results) {
+ match result {
+ Ok(output) => {
+ written_paths.push(plan.base_file_path.clone());
+ if let Some(ranges) = output.ranges {
+ stats_updates.push(crate::write::metadata::StatsFileUpdate {
+ partition_path: plan.partition_path.clone(),
+ file_name: plan.file_name.clone(),
+ is_deleted: false,
+ ranges,
+ });
+ }
+ write_stats
+ .entry(plan.partition_path.clone())
+ .or_default()
+ .push(HoodieWriteStat {
+ file_id: Some(plan.file_id.clone()),
+ path: Some(plan.base_file_path.clone()),
+ // Basename only — FileGroup builder joins partition_path + name.
+ base_file: Some(plan.file_name.clone()),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(plan.row_count),
+ num_inserts: Some(plan.row_count),
+ total_write_bytes: Some(output.size),
+ file_size_in_bytes: Some(output.size),
+ partition_path: Some(plan.partition_path.clone()),
+ ..Default::default()
+ });
+ files_mdt.push((
+ plan.partition_path.clone(),
+ plan.file_name.clone(),
+ output.size,
+ ));
+ for key in &plan.chunk_keys {
+ rli_entries.push(RecordIndexEntry {
+ record_key: key.record_key.clone(),
+ partition_path: plan.partition_path.clone(),
+ file_id: plan.file_id.clone(),
+ instant_time_millis: crate::write::metadata::instant_to_epoch_millis_in(
+ &request_instant,
+ &crate::write::metadata::timeline_timezone(&table.hudi_configs),
+ ),
+ is_deleted: false,
+ });
+ }
+ }
+ Err(error) => {
+ if first_error.is_none() {
+ first_error = Some(error);
+ }
+ }
+ }
+ }
+ if let Some(error) = first_error {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+
+ let commit_metadata = HoodieCommitMetadata {
+ version: Some(1),
+ operation_type: Some("INSERT".to_string()),
+ partition_to_write_stats: Some(write_stats),
+ compacted: Some(false),
+ // Java HoodieCommitMetadata.extraMetadata is non-null; null → NPE in Spark.
+ extra_metadata: Some(HashMap::from([(
+ "schema".to_string(),
+ arrow_schema_to_avro_json(&schema)?,
+ )])),
+ };
+ let commit_bytes = if layout_two {
+ commit_metadata.to_avro_bytes()?
+ } else {
+ commit_metadata.to_json_bytes()?
+ };
+
+ // Crash-consistency order: data files → MDT files → MDT deltacommit →
+ // completed data commit. Finalizing the data timeline last means readers
+ // fence MDT contents by completed data instants. MDT log-file writes are
+ // "work" (unlocked); only completion runs in critical section 2.
+ let mut mdt_stats: HashMap<String, Vec<HoodieWriteStat>> = HashMap::new();
+ if table.is_metadata_table_enabled() {
+ let mdt_result = async {
+ mdt_stats.insert(
+ "files".to_string(),
+ update_files_partition(storage.as_ref(), &request_instant, &files_mdt).await?,
+ );
+ if is_record_index_enabled(table) {
+ let stats =
+ update_record_index(storage.as_ref(), &request_instant, &rli_entries).await?;
+ if !stats.is_empty() {
+ mdt_stats.insert("record_index".to_string(), stats);
+ }
+ }
+ if is_column_stats_enabled(table) && !stats_updates.is_empty() {
+ let stats = update_column_stats_partitions(
+ table,
+ &request_instant,
+ &stats_updates,
+ schema.as_ref(),
+ &[],
+ )
+ .await?;
+ mdt_stats.extend(stats);
+ }
+ Ok::<(), crate::error::CoreError>(())
+ }
+ .await;
+ if let Err(error) = mdt_result {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ }
+
+ // Critical section 2: complete the action (MDT deltacommit + data commit
+ // with a completion time minted under the lock), then bookkeeping.
+ let cs2 = lock.lock().await?;
+ if table.is_metadata_table_enabled()
+ && let Err(error) = crate::write::metadata::write_metadata_commit(
+ storage.as_ref(),
+ &request_instant,
+ mdt_stats,
+ )
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error);
+ }
+ let completion = if layout_two {
+ Some(generate_instant_time().await)
+ } else {
+ None
+ };
+ let instant = Instant::new_completed(request_instant.clone(), action, completion)?;
+ let commit_relative_path = instant.relative_path_with_base(&timeline_dir)?;
+ if let Err(error) = storage
+ .put_file_if_absent(&commit_relative_path, commit_bytes)
+ .await
+ {
+ for path in &written_paths {
+ let _ = storage.delete_file(path).await;
+ }
+ return Err(error.into());
+ }
+ // The commit is durable from here: post-commit maintenance (markers,
+ // archival) and handle refresh are best-effort — a transient failure must
+ // not turn a committed write into an Err the caller would retry.
+ let _ =
+ crate::write::post_complete_bookkeeping(table, storage.as_ref(), &request_instant).await;
+ drop(cs2);
+
+ let _ = table.timeline.reload_completed_commits().await;
+ table.file_system_view.clear_cache();
+
+ Ok(AppendResult {
+ instant: request_instant,
+ commit_relative_path,
+ base_file_path: primary_base_path,
+ num_rows,
+ })
+}
+
+fn take_rows(batch: &RecordBatch, indices: &[u32]) -> Result<RecordBatch> {
+ let indices = UInt32Array::from(indices.to_vec());
+ let columns = batch
+ .columns()
+ .iter()
+ .map(|column| take(column.as_ref(), &indices, None))
+ .collect::<std::result::Result<Vec<_>, _>>()?;
+ RecordBatch::try_new(batch.schema(), columns).map_err(Into::into)
+}
+
+fn ensure_insert_allowed(table: &Table) -> Result<()> {
+ let strategy: String = table
+ .hudi_configs
+ .get_or_default(RecordMergeStrategy)
+ .into();
+ // Insert/append is valid for both append-only and upsertable (commit/event-time) tables.
+ match RecordMergeStrategyValue::from_str(&strategy)? {
+ RecordMergeStrategyValue::AppendOnly | RecordMergeStrategyValue::OverwriteWithLatest => {
+ Ok(())
+ }
+ }
+}
+
+fn ensure_append_only_strategy(table: &Table) -> Result<()> {
+ let strategy: String = table
+ .hudi_configs
+ .get_or_default(RecordMergeStrategy)
+ .into();
+ if RecordMergeStrategyValue::from_str(&strategy)? != RecordMergeStrategyValue::AppendOnly {
+ return Err(CoreError::Unsupported(
+ "append_only requires hoodie.record.merge.mode=append_only".to_string(),
+ ));
+ }
+ Ok(())
+}
+
+pub(crate) fn ensure_copy_on_write(table: &Table) -> Result<()> {
+ let table_type: String = table.hudi_configs.get(TableType)?.into();
+ if TableTypeValue::from_str(&table_type)? != TableTypeValue::CopyOnWrite {
+ return Err(CoreError::Unsupported(format!(
+ "append is currently only supported for COPY_ON_WRITE tables. Found '{table_type}'"
+ )));
+ }
+ Ok(())
+}
+
+fn ensure_supported_table_type(table: &Table) -> Result<()> {
+ let table_type: String = table.hudi_configs.get(TableType)?.into();
+ match TableTypeValue::from_str(&table_type)? {
+ TableTypeValue::CopyOnWrite | TableTypeValue::MergeOnRead => Ok(()),
+ }
+}
+
+pub(crate) fn is_layout_two(table: &Table) -> bool {
+ let table_version: isize = match table.hudi_configs.get(TableVersion) {
+ Ok(v) => v.into(),
+ Err(_) => return false,
+ };
+ let layout_version: isize = match table.hudi_configs.get(TimelineLayoutVersion) {
+ Ok(v) => v.into(),
+ Err(_) => return false,
+ };
+ table_version >= 8 && layout_version == 2
+}
+
+pub(crate) fn timeline_dir(table: &Table) -> String {
+ if is_layout_two(table) {
+ let timeline_path: String = table.hudi_configs.get_or_default(TimelinePath).into();
+ format!("{HUDI_METADATA_DIR}/{timeline_path}")
+ } else {
+ HUDI_METADATA_DIR.to_string()
+ }
+}
+
+/// Java `HoodieTimeGeneratorConfig` infers a 1 ms max expected clock skew when
+/// the (default) `InProcessLockProvider` is used — the single-writer case.
+const MAX_EXPECTED_CLOCK_SKEW_MS: u64 = 1;
+
+/// Skew-adjusting monotonic instant time generator (Java
+/// `SkewAdjustingTimeGenerator` + `HoodieInstantTimeGenerator.createNewInstantTime`).
+///
+/// Under a process-wide lock: capture the clock, wait out the max expected
+/// clock skew, and return the captured time — retrying (never bumping +1) until
+/// it exceeds every previously minted instant. Used for both requested and
+/// completion timestamps, so completion times are strictly greater than the
+/// requested times minted before them. Format `yyyyMMddHHmmssSSS` (UTC).
+/// Whether instants are formatted in local time (Java's
+/// `HoodieTimelineTimeZone.LOCAL`, the Hudi default) or UTC. Process-wide,
+/// mirroring `HoodieInstantTimeGenerator.setCommitTimeZone`; set from the
+/// table's `hoodie.table.timeline.timezone` when a table is created or
+/// loaded for writes. Spark writers mint LOCAL-time instants by default, so
+/// matching the table's declared zone keeps mixed-writer timelines ordered.
+static COMMIT_TIME_IS_LOCAL: std::sync::atomic::AtomicBool =
+ std::sync::atomic::AtomicBool::new(true);
+
+pub(crate) fn set_commit_timezone(timezone: &str) {
+ COMMIT_TIME_IS_LOCAL.store(
+ !timezone.eq_ignore_ascii_case("utc"),
+ std::sync::atomic::Ordering::Release,
+ );
+}
+
+pub(crate) async fn generate_instant_time() -> String {
+ static TIME_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
+ let _guard = TIME_LOCK.lock().await;
+ loop {
+ let now = Utc::now();
+ tokio::time::sleep(std::time::Duration::from_millis(MAX_EXPECTED_CLOCK_SKEW_MS)).await;
+ let candidate_millis = now.timestamp_millis();
+ // Store is race-free: all mints serialize through TIME_LOCK.
+ if candidate_millis > LAST_EPOCH_MILLIS.load(Ordering::Acquire) {
+ LAST_EPOCH_MILLIS.store(candidate_millis, Ordering::Release);
+ let format = "%Y%m%d%H%M%S%3f";
+ return if COMMIT_TIME_IS_LOCAL.load(std::sync::atomic::Ordering::Acquire) {
+ now.with_timezone(&chrono::Local).format(format).to_string()
+ } else {
+ now.format(format).to_string()
+ };
+ }
+ }
+}
+
+/// Default max base-file size (Java `hoodie.parquet.max.file.size` = 120 MiB).
+pub(crate) const DEFAULT_MAX_FILE_SIZE_BYTES: i64 = 120 * 1024 * 1024;
+/// Default record size estimate used for insert splits.
+pub(crate) const DEFAULT_RECORD_SIZE_ESTIMATE: i64 = 1024;
+
+pub(crate) fn max_records_per_file(table: &Table) -> usize {
+ let max_file_size = table
+ .hudi_configs
+ .as_options()
+ .get("hoodie.parquet.max.file.size")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_MAX_FILE_SIZE_BYTES)
+ .max(1);
+ let record_size = table
+ .hudi_configs
+ .as_options()
+ .get("hoodie.copyonwrite.record.size.estimate")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_RECORD_SIZE_ESTIMATE)
+ .max(1);
+ ((max_file_size / record_size) as usize).max(1)
+}
+
+/// Parquet `WriterProperties` resolved from `hoodie.parquet.*` configs.
+pub(crate) fn parquet_writer_props(table: &Table) -> parquet::file::properties::WriterProperties {
+ use parquet::basic::{Compression, GzipLevel, ZstdLevel};
+ use parquet::file::properties::WriterProperties;
+
+ let options = table.hudi_configs.as_options();
+ let page_size = options
+ .get("hoodie.parquet.page.size")
+ .and_then(|v| v.parse::<usize>().ok())
+ .unwrap_or(1024 * 1024)
+ .max(1);
+ let block_size = options
+ .get("hoodie.parquet.block.size")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_MAX_FILE_SIZE_BYTES)
+ .max(1);
+ let record_size = options
+ .get("hoodie.copyonwrite.record.size.estimate")
+ .and_then(|v| v.parse::<i64>().ok())
+ .unwrap_or(DEFAULT_RECORD_SIZE_ESTIMATE)
+ .max(1);
+ // parquet-rs row-group limit is row-count; approximate Java's byte block size.
+ let max_row_group_rows = ((block_size / record_size) as usize).max(1);
+ let dictionary_enabled = options
+ .get("hoodie.parquet.dictionary.enabled")
+ .map(|v| v.eq_ignore_ascii_case("true"))
+ .unwrap_or(true);
+ let codec = options
+ .get("hoodie.parquet.compression.codec")
+ .map(|s| s.as_str())
+ .unwrap_or("zstd");
+ let compression = match codec.to_ascii_lowercase().as_str() {
+ "uncompressed" | "none" => Compression::UNCOMPRESSED,
+ "snappy" => Compression::SNAPPY,
+ "gzip" | "gz" => Compression::GZIP(GzipLevel::try_new(6).unwrap_or_default()),
+ "lz4" => Compression::LZ4,
+ "brotli" => Compression::BROTLI(Default::default()),
+ // Default zstd — intentional vs Java's gzip; honor hoodie.parquet.compression.codec.
+ _ => Compression::ZSTD(ZstdLevel::try_new(3).unwrap_or_default()),
+ };
+
+ WriterProperties::builder()
+ .set_compression(compression)
+ .set_dictionary_enabled(dictionary_enabled)
+ .set_data_page_size_limit(page_size)
+ .set_dictionary_page_size_limit(page_size)
+ .set_max_row_group_row_count(Some(max_row_group_rows))
+ .build()
+}
+
+/// Encode batches to parquet bytes with pre-resolved properties (safe to run
+/// on a blocking thread with owned inputs).
+pub(crate) fn write_parquet_bytes_with_props(
+ props: parquet::file::properties::WriterProperties,
+ batches: &[RecordBatch],
+) -> Result<Vec<u8>> {
+ let cursor = Cursor::new(Vec::new());
+ let mut writer = ArrowWriter::try_new(cursor, batches[0].schema(), Some(props))?;
+ for batch in batches {
+ writer.write(batch)?;
+ }
+ let cursor = writer.into_inner()?;
+ Ok(cursor.into_inner())
+}
+
+/// Arrow→Avro schema JSON for `HoodieCommitMetadata.extraMetadata.schema`.
+///
+/// Unmapped Arrow types error out — silent `"string"` fallback corrupts Spark's
+/// TableSchemaResolver (Parquet types diverge from commit schema).
+pub(crate) fn arrow_schema_to_avro_json(schema: &arrow_schema::Schema) -> Result<String> {
+ let fields: Vec<String> = schema
+ .fields()
+ .iter()
+ .enumerate()
+ .map(|(index, f)| {
+ ensure_legal_avro_name(f.name())?;
+ // Field INDEX, not name: a name-derived path is not injective
+ // (`a.b` and `a_b` collide), and Avro refuses a redefined record.
+ let avro_type = arrow_type_to_avro_type_json_named(
+ f.data_type(),
+ &format!("hoodie_record_f{index}"),
+ )?;
+ let ty = if f.is_nullable() {
+ format!("[\"null\",{avro_type}]")
+ } else {
+ avro_type
+ };
+ Ok(format!(
+ "{{\"name\":\"{}\",\"type\":{}}}",
+ f.name().replace('\"', "\\\""),
+ ty
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+ Ok(format!(
+ "{{\"type\":\"record\",\"name\":\"hoodie_record\",\"fields\":[{}]}}",
+ fields.join(",")
+ ))
+}
+
+/// Avro field names must match [A-Za-z_][A-Za-z0-9_]*. A name that does not is
+/// rejected rather than renamed: renaming would diverge from the parquet column
+/// and silently mislabel the data, while an unparseable commit schema makes the
+/// table unreadable by every engine (including this one).
+fn ensure_legal_avro_name(name: &str) -> Result<()> {
+ let mut chars = name.chars();
+ let legal = chars
+ .next()
+ .is_some_and(|c| c.is_ascii_alphabetic() || c == '_')
+ && chars.all(|c| c.is_ascii_alphanumeric() || c == '_');
+ if legal {
+ return Ok(());
+ }
+ Err(CoreError::Unsupported(format!(
+ "column name '{name}' is not a legal Avro name ([A-Za-z_][A-Za-z0-9_]*), \
+ so it cannot be written into Hudi commit metadata"
+ )))
+}
+
+/// `path` makes every nested Avro record name unique: Avro forbids redefining
+/// a named type, so a fixed name breaks any schema with two structs — Java's
+/// `Schema.Parser` rejects the whole commit schema.
+fn arrow_type_to_avro_type_json_named(dt: &arrow_schema::DataType, path: &str) -> Result<String> {
+ use arrow_schema::DataType;
+ match dt {
+ DataType::Boolean => Ok("\"boolean\"".to_string()),
+ DataType::Int8 | DataType::Int16 | DataType::Int32 => Ok("\"int\"".to_string()),
+ DataType::Int64 => Ok("\"long\"".to_string()),
+ DataType::Float32 => Ok("\"float\"".to_string()),
+ DataType::Float64 => Ok("\"double\"".to_string()),
+ DataType::Utf8 | DataType::LargeUtf8 => {
+ Ok("{\"type\":\"string\",\"avro.java.string\":\"String\"}".to_string())
+ }
+ DataType::Binary | DataType::LargeBinary => Ok("\"bytes\"".to_string()),
+ DataType::Date32 => Ok("{\"type\":\"int\",\"logicalType\":\"date\"}".to_string()),
+ DataType::Timestamp(unit, tz) => {
+ // A timezone-less Arrow timestamp is Spark's TimestampNTZType,
+ // which maps to Avro local-timestamp-*; mapping it to the
+ // UTC-adjusted logical type shifts values on read.
+ let local = tz.is_none();
+ let logical = match unit {
+ arrow_schema::TimeUnit::Millisecond if local => "local-timestamp-millis",
+ arrow_schema::TimeUnit::Microsecond if local => "local-timestamp-micros",
+ arrow_schema::TimeUnit::Millisecond => "timestamp-millis",
+ arrow_schema::TimeUnit::Microsecond => "timestamp-micros",
+ other => {
+ return Err(CoreError::Unsupported(format!(
+ "commit schema does not support Arrow timestamp unit {other:?}"
+ )));
+ }
+ };
+ Ok(format!(
+ "{{\"type\":\"long\",\"logicalType\":\"{logical}\"}}"
+ ))
+ }
+ DataType::Decimal128(precision, scale) | DataType::Decimal256(precision, scale) => {
+ Ok(format!(
+ "{{\"type\":\"bytes\",\"logicalType\":\"decimal\",\"precision\":{precision},\"scale\":{scale}}}"
+ ))
+ }
+ DataType::List(field) | DataType::LargeList(field) => {
+ let item =
+ arrow_type_to_avro_type_json_named(field.data_type(), &format!("{path}_item"))?;
+ let item = if field.is_nullable() {
+ format!("[\"null\",{item}]")
+ } else {
+ item
+ };
+ Ok(format!("{{\"type\":\"array\",\"items\":{item}}}"))
+ }
+ DataType::Struct(fields) => {
+ let nested: Vec<String> = fields
+ .iter()
+ .enumerate()
+ .map(|(index, f)| {
+ ensure_legal_avro_name(f.name())?;
+ let avro_type = arrow_type_to_avro_type_json_named(
+ f.data_type(),
+ &format!("{path}f{index}"),
+ )?;
+ let ty = if f.is_nullable() {
+ format!("[\"null\",{avro_type}]")
+ } else {
+ avro_type
+ };
+ Ok(format!(
+ "{{\"name\":\"{}\",\"type\":{}}}",
+ f.name().replace('\"', "\\\""),
+ ty
+ ))
+ })
+ .collect::<Result<Vec<_>>>()?;
+ Ok(format!(
+ "{{\"type\":\"record\",\"name\":\"{path}_record\",\"fields\":[{}]}}",
+ nested.join(",")
+ ))
+ }
+ other => Err(CoreError::Unsupported(format!(
+ "cannot map Arrow type {other:?} to Avro for commit metadata schema"
+ ))),
+ }
+}
+
+async fn ensure_append_schema_matches_table(
+ table: &Table,
+ batch_schema: &arrow_schema::Schema,
+) -> Result<()> {
+ if table.timeline.completed_commits.is_empty() {
+ return Ok(());
+ }
+ let table_schema = table.get_schema().await?;
+ let batch_data = strip_meta_fields_from_schema(batch_schema);
+ if schemas_equal_by_fields(&table_schema, &batch_data) {
+ return Ok(());
+ }
+ Err(CoreError::Write(format!(
+ "append batch schema does not match table schema (table fields={:?}, batch fields={:?})",
+ table_schema
+ .fields()
+ .iter()
+ .map(|f| (f.name().clone(), f.data_type().clone()))
+ .collect::<Vec<_>>(),
+ batch_data
+ .fields()
+ .iter()
+ .map(|f| (f.name().clone(), f.data_type().clone()))
+ .collect::<Vec<_>>(),
+ )))
+}
+
+pub(crate) fn strip_meta_fields_from_schema(schema: &arrow_schema::Schema) -> arrow_schema::Schema {
+ let fields: Vec<_> = schema
+ .fields()
+ .iter()
+ .filter(|f| !f.name().starts_with("_hoodie_"))
Review Comment:
Fixed in 30342034 and d6c1e48c: `_hoodie_is_deleted` is a data column and marks deletes on upsert and on copy-on-write update.
##########
crates/core/src/table/mod.rs:
##########
@@ -201,17 +200,27 @@ fn instant_time_minus_one(instant_time: &str) -> String {
}
impl Table {
- /// Get or initialize the cached metadata table instance.
+ /// Open a fresh metadata-table handle from storage (no instance cache).
///
- /// Returns `Ok(&Table)` if metadata table is successfully created or was already cached.
- /// The instance is created once and reused across all subsequent calls.
- pub(crate) async fn get_or_init_metadata_table(&self) -> Result<&Table> {
- self.cached_metadata_table
- .get_or_try_init(|| async {
- log::debug!("Initializing cached metadata table instance");
- self.new_metadata_table().await
- })
- .await
+ /// Writers and readers reload the data timeline first, then open MDT relative
+ /// to that view. Commit locking / conflict checks are deferred to a later
+ /// concurrency-control project.
+ pub(crate) async fn get_or_init_metadata_table(&self) -> Result<Table> {
+ self.new_metadata_table().await
+ }
+
+ /// Reload the active timeline and drop listing caches before a write plans.
+ pub(crate) async fn reload_timeline_for_write(&mut self) -> Result<()> {
+ crate::write::ensure_writable_table_version(self)?;
+ // Instants must be minted in the table's declared timeline timezone
+ // (Spark writers default to LOCAL); set before any instant is minted.
+ crate::write::set_commit_timezone(&self.timezone());
+ // Eager rollback of crashed writes (Java rollbackFailedWrites) before
+ // any new instant is minted.
+ crate::write::rollback::rollback_failed_writes(self).await?;
Review Comment:
Kept as a single-writer limitation and documented in b6cfdb86 (`write/lock.rs` and the writer design doc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4404757908
##########
crates/core/src/index/record.rs:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Record-level index backed by the metadata table `record_index` partition.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use crate::Result;
+use crate::error::CoreError;
+use crate::file_group::log_file::scanner::{LogFileScanner, ScanResult};
+use crate::hfile::{HFileReader, HFileRecord};
+use crate::index::{HoodieIndex, HoodieKey, RecordLocation};
+use crate::metadata::table::encode::decode_record_index_entry;
+use crate::metadata::table::records::MetadataPartitionType;
+use crate::storage::Storage;
+use crate::table::Table;
+use crate::timeline::selector::InstantRange;
+use crate::write::metadata::epoch_millis_to_instant;
+
+/// Metadata-table record-level index (global: key = record key).
+#[derive(Clone, Debug, Default)]
+pub struct RecordIndex;
+
+impl HoodieIndex for RecordIndex {
+ async fn tag_location(
+ &self,
+ table: &Table,
+ keys: &[HoodieKey],
+ ) -> Result<HashMap<HoodieKey, Option<RecordLocation>>> {
+ let mut locations = keys
+ .iter()
+ .cloned()
+ .map(|key| (key, None))
+ .collect::<HashMap<_, _>>();
+ if keys.is_empty() {
+ return Ok(locations);
+ }
+
+ let storage = table.file_system_view.storage.clone();
+ let merged = load_record_index_map(table, storage).await?;
+ for key in keys {
+ if let Some(entry) = merged.get(&key.record_key) {
+ locations.insert(
+ key.clone(),
+ Some(RecordLocation {
+ file_id: entry.file_id.clone(),
+ instant_time: epoch_millis_to_instant(entry.instant_time_millis),
+ partition_path: entry.partition_path.clone(),
+ }),
+ );
+ }
+ }
+ Ok(locations)
+ }
+}
+
+struct LoadedLocation {
+ file_id: String,
+ partition_path: String,
+ instant_time_millis: i64,
+}
+
+async fn load_record_index_map(
+ table: &Table,
+ storage: Arc<Storage>,
+) -> Result<HashMap<String, LoadedLocation>> {
+ let partition = MetadataPartitionType::RecordIndex.partition_name();
+ // Slice-resolved: only each file group's LIVE slice (multi-base aware
+ // after a Spark MDT compaction), logs in version order.
+ let mut base_paths = Vec::new();
+ let mut log_paths = Vec::new();
+ for (_, base, logs) in
+ crate::write::metadata::mdt_partition_latest_slices(storage.as_ref(), partition).await?
+ {
+ base_paths.push(base);
+ log_paths.extend(logs);
+ }
+
+ let mut merged: HashMap<String, Option<LoadedLocation>> = HashMap::new();
+ for base in base_paths {
+ let mut reader = HFileReader::open(storage.as_ref(), &base)
+ .await
+ .map_err(|e| CoreError::HFile(format!("failed to open record_index base: {e:?}")))?;
+ let records = reader
+ .collect_records()
+ .map_err(|e| CoreError::HFile(format!("failed to read record_index base: {e:?}")))?;
+ apply_hfile_records(&mut merged, &records)?;
+ }
+ if !log_paths.is_empty() {
+ // Only trust log blocks whose instant completed on the data timeline
+ // (Java getValidInstantTimestamps) — orphan MDT commits are skipped.
+ let valid = table.valid_metadata_instants().await?;
+ let scanner = LogFileScanner::new(table.hudi_configs.clone(), storage);
+ let range = InstantRange::exact_match(valid, "UTC");
+ match scanner.scan(log_paths, &range).await? {
+ ScanResult::HFileRecords(records) => apply_hfile_records(&mut merged, &records)?,
+ ScanResult::Empty => {}
+ ScanResult::RecordBatches(_) => {
+ return Err(CoreError::MetadataTable(
+ "record_index logs must contain HFile data blocks".to_string(),
Review Comment:
Fixed in 20e317bb: record_index logs holding only delete blocks read as tombstones.
##########
crates/core/src/write/metadata.rs:
##########
@@ -0,0 +1,1320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Metadata table bootstrap and files / record_index / stats partition updates.
+
+use std::collections::{BTreeMap, HashMap};
+
+use crate::Result;
+use crate::file_group::log_file::scanner::{LogFileScanner, ScanResult};
+use crate::file_group::log_file::writer::LogFileWriter;
+use crate::file_group::log_file::{BlockMetadataKey, BlockType};
+use crate::hfile::{HFileReader, HFileWriter};
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::table::column_stats::{
+ ColumnRangeStats, aggregate_partition_stats, column_stats_for_file, column_stats_tombstones,
+ columns_to_index,
+};
+use crate::metadata::table::encode::{
+ ColumnStatsMetadata, FilesMetadataEntry, RecordIndexEntry, StatsIndexVersion,
+ decode_column_stats_entry, encode_all_partitions, encode_column_stats, encode_files_record,
+ encode_partition_stats, encode_record_index_entry, files_metadata_avro_schema_json,
+ hoodie_metadata_schema_json, record_index_metadata_avro_schema_json,
+};
+use crate::metadata::table::hash::{column_stats_index_key, partition_stats_index_key};
+use crate::metadata::table::records::{
+ FilesPartitionRecord, MetadataPartitionType, MetadataRecordType,
+};
+use crate::storage::Storage;
+use crate::table::{ReadOptions, Table};
+use crate::timeline::instant::{Action, Instant};
+use crate::timeline::selector::InstantRange;
+use arrow_schema::Schema;
+
+const METADATA_BASE: &str = ".hoodie/metadata";
+const FILES_FILE_ID: &str = "files-0000-0";
+/// Java `hoodie.metadata.record.index.min.filegroup.count` default.
+pub(crate) const DEFAULT_RLI_NUM_FILE_GROUPS: usize = 10;
+/// Java `hoodie.metadata.index.column.stats.file.group.count` default.
+pub(crate) const DEFAULT_COLUMN_STATS_NUM_FILE_GROUPS: usize = 2;
+/// Java `hoodie.metadata.index.partition.stats.file.group.count` default.
+pub(crate) const DEFAULT_PARTITION_STATS_NUM_FILE_GROUPS: usize = 1;
+
+fn column_stats_file_id(shard: usize) -> String {
+ format!("col-stats-{shard:04}-0")
+}
+
+fn partition_stats_file_id(shard: usize) -> String {
+ format!("partition-stats-{shard:04}-0")
+}
+
+/// A parquet (or deleted) data file whose column ranges should update MDT stats.
+#[derive(Debug, Clone)]
+pub struct StatsFileUpdate {
+ pub partition_path: String,
+ pub file_name: String,
+ pub is_deleted: bool,
+ /// Footer-derived ranges for adds; ignored when `is_deleted`.
+ pub ranges: Vec<ColumnRangeStats>,
+}
+
+fn record_index_file_id(shard: usize) -> String {
+ format!("record-index-{shard:04}-0")
+}
+
+/// Java `HoodieTableMetadataUtil.mapRecordKeyToFileGroupIndex` (31-hash, signed i32).
+pub(crate) fn map_record_key_to_file_group_index(
+ record_key: &str,
+ num_file_groups: usize,
+) -> usize {
+ if num_file_groups <= 1 {
+ return 0;
+ }
+ let mut h: i32 = 0;
+ // Java's String.hashCode iterates UTF-16 code units (charAt), not Unicode
+ // scalars; supplementary-plane keys must hash to the same shard as Java.
+ for unit in record_key.encode_utf16() {
+ h = h.wrapping_mul(31).wrapping_add(unit as i32);
+ }
+ (h.unsigned_abs() as usize) % num_file_groups
+}
+
+/// Java `HoodieBackedTableMetadataWriter.generateUniqueInstantTime` on a fresh table:
+/// `SOLO_COMMIT_TIMESTAMP` plus `offset` millis, zero-padded to 17 digits.
+fn bootstrap_instant(offset: usize) -> String {
+ format!("{offset:017}")
+}
+
+/// Create the metadata table properties and its initial partitions.
+///
+/// Each MDT partition is initialized in its own deltacommit with a unique
+/// instant (`00000000000000000`, `...001`, ...) in Java's enum order:
+/// files, column_stats, record_index, partition_stats.
+pub async fn bootstrap_metadata_table(
+ data_table_storage: &Storage,
+ table_name: &str,
+ table_version: isize,
+ record_index_enabled: bool,
+ column_stats_enabled: bool,
+ partition_stats_enabled: bool,
+) -> Result<()> {
+ let mdt_name = format!("{table_name}_metadata");
+ let checksum = crc32(format!(".{mdt_name}").as_bytes());
+ // The MDT's table version follows the data table's (Java initializes the
+ // MDT meta client with the data write config's version).
+ let properties = format!(
+ "# Generated by hudi-rs\n\
+ hoodie.table.name={mdt_name}\n\
+ hoodie.table.type=MERGE_ON_READ\n\
+ hoodie.table.base.file.format=HFILE\n\
+ hoodie.table.recordkey.fields=key\n\
+ hoodie.populate.meta.fields=false\n\
+ hoodie.table.version={table_version}\n\
+ hoodie.table.initial.version={table_version}\n\
+ hoodie.timeline.layout.version=2\n\
+ hoodie.timeline.path=timeline\n\
+ hoodie.timeline.history.path=history\n\
+ hoodie.archivelog.folder=history\n\
+ hoodie.table.keygenerator.type=HOODIE_TABLE_METADATA\n\
+ hoodie.compaction.payload.class=org.apache.hudi.metadata.HoodieMetadataPayload\n\
+ hoodie.record.merge.mode=CUSTOM\n\
+ hoodie.record.merge.strategy.id=00000000-0000-0000-0000-000000000000\n\
+ hoodie.datasource.write.drop.partition.columns=false\n\
+ hoodie.table.checksum={checksum}\n"
+ );
+ data_table_storage
+ .put_file(
+ &format!("{METADATA_BASE}/.hoodie/hoodie.properties"),
+ properties.into_bytes(),
+ )
+ .await?;
+ data_table_storage
+ .put_file(
+ &format!("{METADATA_BASE}/.hoodie/timeline/.keep"),
+ b"".as_slice(),
+ )
+ .await?;
+
+ // Java initializeFilesPartition on an empty table writes only the
+ // __all_partitions__ record with an empty partition list — no "." entry.
+ let files_instant = bootstrap_instant(0);
+ let all_partitions = encode_all_partitions(Vec::new())?;
+ let files_base_name = format!("{FILES_FILE_ID}_0-0-0_{files_instant}.hfile");
+ let hfile = HFileWriter::write(
+ &[(
+ FilesPartitionRecord::ALL_PARTITIONS_KEY.to_string(),
+ all_partitions,
+ )],
+ BTreeMap::from([(
+ "schema".to_string(),
+ files_metadata_avro_schema_json()?.as_bytes().to_vec(),
+ )]),
+ )
+ .map_err(|error| crate::error::CoreError::HFile(error.to_string()))?;
+ let files_base_size = hfile.len() as i64;
+ crate::write::ensure_partition_metadata(
+ data_table_storage,
+ &format!("{METADATA_BASE}/files"),
+ &files_instant,
+ )
+ .await?;
+ data_table_storage
+ .put_file(&format!("{METADATA_BASE}/files/{files_base_name}"), hfile)
+ .await?;
+ write_metadata_commit(
+ data_table_storage,
+ &files_instant,
+ HashMap::from([(
+ "files".to_string(),
+ vec![HoodieWriteStat {
+ file_id: Some(FILES_FILE_ID.to_string()),
+ path: Some(format!("files/{files_base_name}")),
+ base_file: Some(files_base_name),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(1),
+ total_write_bytes: Some(files_base_size),
+ file_size_in_bytes: Some(files_base_size),
+ partition_path: Some("files".to_string()),
+ ..Default::default()
+ }],
+ )]),
+ )
+ .await?;
+
+ // Remaining partitions in Java MetadataPartitionType enum order, each with
+ // the next unique bootstrap instant and its own deltacommit.
+ let mut next_offset = 1usize;
+ if column_stats_enabled {
+ bootstrap_empty_hfile_partition(
+ data_table_storage,
+ &bootstrap_instant(next_offset),
+ MetadataPartitionType::ColumnStats.partition_name(),
+ DEFAULT_COLUMN_STATS_NUM_FILE_GROUPS,
+ &column_stats_file_id,
+ hoodie_metadata_schema_json()?,
+ )
+ .await?;
+ next_offset += 1;
+ }
+ if record_index_enabled {
+ bootstrap_empty_hfile_partition(
+ data_table_storage,
+ &bootstrap_instant(next_offset),
+ MetadataPartitionType::RecordIndex.partition_name(),
+ DEFAULT_RLI_NUM_FILE_GROUPS,
+ &record_index_file_id,
+ record_index_metadata_avro_schema_json()?,
+ )
+ .await?;
+ next_offset += 1;
+ }
+ if partition_stats_enabled {
+ bootstrap_empty_hfile_partition(
+ data_table_storage,
+ &bootstrap_instant(next_offset),
+ MetadataPartitionType::PartitionStats.partition_name(),
+ DEFAULT_PARTITION_STATS_NUM_FILE_GROUPS,
+ &partition_stats_file_id,
+ hoodie_metadata_schema_json()?,
+ )
+ .await?;
+ }
+ Ok(())
+}
+
+async fn bootstrap_empty_hfile_partition(
+ storage: &Storage,
+ instant: &str,
+ partition_name: &str,
+ num_file_groups: usize,
+ file_id_fn: &dyn Fn(usize) -> String,
+ schema_json: &str,
+) -> Result<()> {
+ crate::write::ensure_partition_metadata(
+ storage,
+ &format!("{METADATA_BASE}/{partition_name}"),
+ instant,
+ )
+ .await?;
+ let mut stats = Vec::with_capacity(num_file_groups);
+ for shard in 0..num_file_groups {
+ let file_id = file_id_fn(shard);
+ let base_name = format!("{file_id}_0-0-0_{instant}.hfile");
+ let empty = HFileWriter::write(
+ &[],
+ BTreeMap::from([("schema".to_string(), schema_json.as_bytes().to_vec())]),
+ )
+ .map_err(|error| crate::error::CoreError::HFile(error.to_string()))?;
+ let size = empty.len() as i64;
+ storage
+ .put_file(
+ &format!("{METADATA_BASE}/{partition_name}/{base_name}"),
+ empty,
+ )
+ .await?;
+ stats.push(HoodieWriteStat {
+ file_id: Some(file_id),
+ path: Some(format!("{partition_name}/{base_name}")),
+ base_file: Some(base_name),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(0),
+ total_write_bytes: Some(size),
+ file_size_in_bytes: Some(size),
+ partition_path: Some(partition_name.to_string()),
+ ..Default::default()
+ });
+ }
+ write_metadata_commit(
+ storage,
+ instant,
+ HashMap::from([(partition_name.to_string(), stats)]),
+ )
+ .await
+}
+
+/// Append a files-partition HFile data block to the metadata table log file.
+///
+/// Returns the `files` partition write stats for the caller's single MDT
+/// deltacommit (see [`write_metadata_commit`]).
+pub async fn update_files_partition(
+ data_storage: &Storage,
+ instant: &str,
+ files: &[(String, String, i64)],
+) -> Result<Vec<HoodieWriteStat>> {
+ let entries = files
+ .iter()
+ .map(|(partition, file_name, size)| (partition.clone(), file_name.clone(), *size, false))
+ .collect::<Vec<_>>();
+ update_files_partition_entries(data_storage, instant, &entries).await
+}
+
+/// Apply file additions and deletions to the metadata table files partition.
+///
+/// Returns the `files` partition write stats for the caller's single MDT
+/// deltacommit (see [`write_metadata_commit`]).
+pub async fn update_files_partition_entries(
+ data_storage: &Storage,
+ instant: &str,
+ files: &[(String, String, i64, bool)],
+) -> Result<Vec<HoodieWriteStat>> {
+ let mut by_partition: BTreeMap<String, Vec<FilesMetadataEntry>> = BTreeMap::new();
+ for (partition, file_name, size, is_deleted) in files {
+ let partition = if partition.is_empty() {
+ FilesPartitionRecord::NON_PARTITIONED_NAME.to_string()
+ } else {
+ partition.clone()
+ };
+ by_partition
+ .entry(partition)
+ .or_default()
+ .push(FilesMetadataEntry {
+ name: file_name.clone(),
+ size: *size,
+ is_deleted: *is_deleted,
+ });
+ }
+ let partitions = by_partition.keys().cloned().collect::<Vec<_>>();
+ let all_partition_entries: Vec<FilesMetadataEntry> = partitions
+ .iter()
+ .map(|name| FilesMetadataEntry {
+ name: name.clone(),
+ size: 0,
+ is_deleted: false,
+ })
+ .collect();
+ let mut entries = vec![(
+ FilesPartitionRecord::ALL_PARTITIONS_KEY.to_string(),
+ encode_files_record(
+ FilesPartitionRecord::ALL_PARTITIONS_KEY,
+ MetadataRecordType::AllPartitions,
+ all_partition_entries,
+ )?,
+ )];
+ for (partition, files) in by_partition {
+ entries.push((
+ partition.clone(),
+ encode_files_record(&partition, MetadataRecordType::Files, files)?,
+ ));
+ }
+ let schema_json = files_metadata_avro_schema_json()?.to_string();
+ let hfile = HFileWriter::write(
+ &entries,
+ BTreeMap::from([("schema".to_string(), schema_json.as_bytes().to_vec())]),
+ )
+ .map_err(|error| crate::error::CoreError::HFile(error.to_string()))?;
+ // Java HFile data blocks embed SCHEMA (ordinal 2) alongside INSTANT_TIME.
+ let block = LogFileWriter::write_log_block(
+ BlockType::HfileData,
+ HashMap::from([
+ (BlockMetadataKey::InstantTime, instant.to_string()),
+ (BlockMetadataKey::Schema, schema_json),
+ ]),
+ &hfile,
+ );
+ let log_name = format!(".{FILES_FILE_ID}_{instant}.log.1_0-0-0");
+ let log_rel = format!("files/{log_name}");
+ let log_size = block.len() as i64;
+ data_storage
+ .put_file(&format!("{METADATA_BASE}/{log_rel}"), block)
+ .await?;
+ Ok(vec![HoodieWriteStat {
+ file_id: Some(FILES_FILE_ID.to_string()),
+ path: Some(log_rel),
+ log_files: Some(vec![log_name]),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(entries.len() as i64),
+ total_write_bytes: Some(log_size),
+ file_size_in_bytes: Some(log_size),
+ partition_path: Some("files".to_string()),
+ ..Default::default()
+ }])
+}
+
+/// Append record-index put/delete entries.
+///
+/// Puts are HFile data blocks. Deletes are v3 delete log blocks — never type-5
+/// records with null `recordIndexMetadata`, which NPEs Java's RLI reader.
+///
+/// Returns the `record_index` partition write stats for the caller's single
+/// MDT deltacommit (see [`write_metadata_commit`]).
+pub async fn update_record_index(
+ data_storage: &Storage,
+ instant: &str,
+ entries: &[RecordIndexEntry],
+) -> Result<Vec<HoodieWriteStat>> {
+ if entries.is_empty() {
+ return Ok(Vec::new());
+ }
+ let num_groups = detect_rli_num_file_groups(data_storage).await?;
+ let schema_json = record_index_metadata_avro_schema_json()?.to_string();
+ let mut puts_by_shard: HashMap<usize, Vec<(String, Vec<u8>)>> = HashMap::new();
+ let mut deletes_by_shard: HashMap<usize, Vec<(String, String)>> = HashMap::new();
+ for entry in entries {
+ let shard = map_record_key_to_file_group_index(&entry.record_key, num_groups);
+ if entry.is_deleted {
+ // DeleteRecord.partitionPath is the MDT partition (Java HoodieKey for RLI deletes).
+ deletes_by_shard.entry(shard).or_default().push((
+ entry.record_key.clone(),
+ MetadataPartitionType::RecordIndex
+ .partition_name()
+ .to_string(),
+ ));
+ } else {
+ let bytes = encode_record_index_entry(entry)?;
+ puts_by_shard
+ .entry(shard)
+ .or_default()
+ .push((entry.record_key.clone(), bytes));
+ }
+ }
+
+ let shards: std::collections::BTreeSet<usize> = puts_by_shard
+ .keys()
+ .chain(deletes_by_shard.keys())
+ .copied()
+ .collect();
+ let mut stats = Vec::new();
+ for shard in shards {
+ let file_id = record_index_file_id(shard);
+ let puts = puts_by_shard.remove(&shard).unwrap_or_default();
+ let deletes = deletes_by_shard.remove(&shard).unwrap_or_default();
+ let mut log_bytes = Vec::new();
+ let mut num_writes = 0i64;
+ if !puts.is_empty() {
+ num_writes += puts.len() as i64;
+ let hfile = HFileWriter::write(
+ &puts,
+ BTreeMap::from([("schema".to_string(), schema_json.as_bytes().to_vec())]),
+ )
+ .map_err(|error| crate::error::CoreError::HFile(error.to_string()))?;
+ log_bytes.extend(LogFileWriter::write_log_block(
+ BlockType::HfileData,
+ HashMap::from([
+ (BlockMetadataKey::InstantTime, instant.to_string()),
+ (BlockMetadataKey::Schema, schema_json.clone()),
+ ]),
+ &hfile,
+ ));
+ }
+ if !deletes.is_empty() {
+ num_writes += deletes.len() as i64;
+ log_bytes.extend(crate::write::build_delete_log_block(instant, &deletes, 0)?);
+ }
+ if log_bytes.is_empty() {
+ continue;
+ }
+ let log_name = format!(".{file_id}_{instant}.log.1_0-0-0");
+ let log_rel = format!("record_index/{log_name}");
+ let log_size = log_bytes.len() as i64;
+ data_storage
+ .put_file(&format!("{METADATA_BASE}/{log_rel}"), log_bytes)
+ .await?;
+ stats.push(HoodieWriteStat {
+ file_id: Some(file_id),
+ path: Some(log_rel),
+ log_files: Some(vec![log_name]),
+ prev_commit: Some("null".to_string()),
+ num_writes: Some(num_writes),
+ total_write_bytes: Some(log_size),
+ file_size_in_bytes: Some(log_size),
+ partition_path: Some("record_index".to_string()),
+ ..Default::default()
+ });
+ }
+ Ok(stats)
+}
+
+async fn detect_rli_num_file_groups(storage: &Storage) -> Result<usize> {
+ detect_num_file_groups(
+ storage,
+ MetadataPartitionType::RecordIndex.partition_name(),
+ "record-index-",
+ DEFAULT_RLI_NUM_FILE_GROUPS,
+ )
+ .await
+}
+
+async fn detect_num_file_groups(
Review Comment:
Fixed in 8c36b334: writes are refused when the metadata table has partitions, inflight partitions, or a partitioned record index the writer doesn't maintain (documented in 7d91e592).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #666:
URL: https://github.com/apache/hudi-rs/pull/666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): spark-parity writer: all write operations, table types, query types, markers, archival, locks, MDT indexes, parity harness [hudi-rs]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#issuecomment-5321570394
**For reviewers: what the API looks like**
The full runnable tour is in [`crates/core/examples/table_writes.rs`](https://github.com/vinothchandar/hudi-rs/blob/feat/table-writes/crates/core/examples/table_writes.rs) — `cargo run -p hudi-core --example table_writes`. The short version:
```rust
// Create — config lives on the builder; MDT + record-level index on by default.
let mut table = Table::create("/tmp/trips")
.with_table_name("trips")
.with_table_type(TableTypeValue::CopyOnWrite) // or MergeOnRead
.with_record_key_fields(["id"]) // omit → auto-generated keys
.with_partition_fields(["city"]) // hive-style paths
.with_ordering_fields(["ts"]) // event-time; omit → commit-time
.create()
.await?;
// All verbs take Arrow RecordBatches.
table.append([batch]).await?; // insert-only fast path
table.upsert([batch]).await?; // keyed upsert via the record-level index
table.update("city = 'nyc'", set_fare).await?; // SET columns on rows matching a filter
table.delete("fare > 100").await?; // key predicates → index; others → scoped scan
table.overwrite([batch]).await?; // INSERT_OVERWRITE_TABLE
table.dynamic_partition_overwrite([batch]).await?; // INSERT_OVERWRITE (partitions in input)
// Reads work against the same handle: snapshot, time-travel, incremental.
let rows = table.read(&ReadOptions::new()).await?;
let changes = table
.read(&ReadOptions::new().with_query_type(QueryType::Incremental))
.await?;
```
Everything is `Table`-owned and path-based (no catalog); unsupported combinations (e.g. partial upsert on MOR, custom payload classes) return typed `Unsupported` errors rather than writing something Spark can't read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4404754794
##########
crates/core/src/metadata/table/mod.rs:
##########
@@ -455,6 +469,7 @@ impl Table {
let data_pending = self.timeline.all_pending_instant_times().await?;
let mut valid: HashSet<String> = self.valid_from_completed_data_instants();
+ valid.extend(self.archived_data_instants().await?);
Review Comment:
Fixed in 9c2777c8: metadata-table reads use only the active timeline.
##########
crates/core/src/table/mod.rs:
##########
@@ -201,17 +200,27 @@ fn instant_time_minus_one(instant_time: &str) -> String {
}
impl Table {
- /// Get or initialize the cached metadata table instance.
+ /// Open a fresh metadata-table handle from storage (no instance cache).
///
- /// Returns `Ok(&Table)` if metadata table is successfully created or was already cached.
- /// The instance is created once and reused across all subsequent calls.
- pub(crate) async fn get_or_init_metadata_table(&self) -> Result<&Table> {
- self.cached_metadata_table
- .get_or_try_init(|| async {
- log::debug!("Initializing cached metadata table instance");
- self.new_metadata_table().await
- })
- .await
+ /// Writers and readers reload the data timeline first, then open MDT relative
+ /// to that view. Commit locking / conflict checks are deferred to a later
+ /// concurrency-control project.
+ pub(crate) async fn get_or_init_metadata_table(&self) -> Result<Table> {
Review Comment:
Fixed in 55112538: the metadata-table handle is cached for reads and dropped when the handle writes.
##########
crates/core/src/write/rollback.rs:
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Eager marker-based rollback of failed writes (Java `EAGER` cleaning policy).
+//!
+//! At write start, any pending data instant (fenced but never completed) is
+//! rolled back: its marker-listed base/log files are deleted (tv8+ rollback is
+//! deletion-only), its completed-but-orphaned MDT deltacommit and MDT log
+//! files are removed, a `rollback` instant records the operation, and the
+//! failed instant's fencing files and marker dir are cleaned up.
+
+use std::collections::HashMap;
+
+use crate::Result;
+use crate::metadata::rollback::{
+ HoodieInstantInfo, HoodieRollbackMetadata, HoodieRollbackPartitionMetadata, HoodieRollbackPlan,
+ HoodieRollbackRequest,
+};
+use crate::storage::Storage;
+use crate::table::Table;
+use crate::write::append::{generate_instant_time, is_layout_two, timeline_dir};
+use crate::write::markers::{delete_marker_dir, parse_marker_name, read_markers};
+
+const METADATA_BASE: &str = ".hoodie/metadata";
+
+/// A pending (fenced, never completed) instant on the data timeline.
+#[derive(Debug, Clone)]
+struct PendingInstant {
+ timestamp: String,
+ action: String,
+ /// Fencing file names present on the timeline (to delete after rollback).
+ fencing_files: Vec<String>,
+}
+
+/// Roll back all pending data instants (Java `rollbackFailedWrites`, EAGER).
+///
+/// Single-writer: any pending instant belongs to a crashed previous write.
+/// Rollback instants themselves and MDT-internal instants are never targets.
+pub(crate) async fn rollback_failed_writes(table: &Table) -> Result<()> {
+ let storage = table.file_system_view.storage.clone();
+ let timeline = timeline_dir(table);
+ let pending = find_pending_instants(storage.as_ref(), &timeline).await?;
+ for instant in pending {
+ rollback_instant(table, storage.as_ref(), &timeline, &instant).await?;
+ }
+ Ok(())
+}
+
+/// The action infix of a fencing file name: `{ts}.{action}.requested|inflight`
+/// or the bare COW forms `{ts}.inflight` / `{ts}.requested` (empty action).
+///
+/// Must not depend on which of an instant's fencing files is listed first —
+/// directory listing order is filesystem-specific (ext4 returns hash order,
+/// so `.inflight` can precede `.commit.requested`).
+fn action_from_fencing_name(name: &str, timestamp: &str) -> String {
+ let rest = name.trim_start_matches(timestamp).trim_start_matches('.');
+ match rest {
+ "inflight" | "requested" => String::new(),
+ other => other
+ .trim_end_matches(".requested")
+ .trim_end_matches(".inflight")
+ .to_string(),
+ }
+}
+
+async fn find_pending_instants(
+ storage: &Storage,
+ timeline_dir: &str,
+) -> Result<Vec<PendingInstant>> {
+ let files = match storage.list_files(Some(timeline_dir)).await {
+ Ok(files) => files,
+ Err(crate::storage::error::StorageError::ObjectStoreError(
+ object_store::Error::NotFound { .. },
+ )) => return Ok(Vec::new()),
+ Err(error) => return Err(error.into()),
+ };
+ let mut completed = std::collections::HashSet::new();
+ let mut fenced: HashMap<String, (String, Vec<String>)> = HashMap::new();
+ for file in &files {
+ let name = &file.name;
+ let Some(first) = name.chars().next() else {
+ continue;
+ };
+ if !first.is_ascii_digit() {
+ continue;
+ }
+ let timestamp: String = name.chars().take_while(char::is_ascii_digit).collect();
+ if name.ends_with(".requested") || name.ends_with(".inflight") {
+ let action = action_from_fencing_name(name, ×tamp);
+ let entry = fenced
+ .entry(timestamp)
+ .or_insert_with(|| (action.clone(), Vec::new()));
+ // Prefer the action-bearing name (COW inflight has no action infix).
+ if entry.0.is_empty() && !action.is_empty() {
+ entry.0 = action;
+ }
+ entry.1.push(name.clone());
+ } else {
+ completed.insert(timestamp);
+ }
+ }
+ let mut pending: Vec<PendingInstant> = fenced
+ .into_iter()
+ .filter(|(ts, _)| !completed.contains(ts))
+ // Rollback/compaction-style instants are never rolled back here.
+ .filter(|(_, (action, _))| {
+ matches!(
+ action.as_str(),
+ "commit" | "deltacommit" | "replacecommit" | ""
+ )
+ })
+ .map(|(timestamp, (action, fencing_files))| PendingInstant {
+ timestamp,
+ action: if action.is_empty() {
+ "commit".to_string()
+ } else {
+ action
+ },
+ fencing_files,
+ })
+ .collect();
+ pending.sort_by(|a, b| a.timestamp.cmp(&b.timestamp));
+ Ok(pending)
+}
+
+async fn rollback_instant(
+ table: &Table,
+ storage: &Storage,
+ timeline_dir: &str,
+ pending: &PendingInstant,
+) -> Result<()> {
+ let start_time = std::time::Instant::now();
+ let failed_ts = &pending.timestamp;
+
+ // Markers tell us exactly what the failed write intended to put. A crash
+ // before markers means no data or MDT files were written (markers are
+ // written first), so an empty list still produces a valid rollback.
+ let markers = read_markers(storage, failed_ts).await?;
+ let mut requests_by_partition: HashMap<String, HoodieRollbackRequest> = HashMap::new();
+ let mut partition_meta: HashMap<String, HoodieRollbackPartitionMetadata> = HashMap::new();
+ let mut total_deleted = 0i32;
+ for marker in &markers {
+ let Some((file_path, _io_type)) = parse_marker_name(marker) else {
+ continue;
+ };
+ let (partition, file_name) = match file_path.rsplit_once('/') {
+ Some((partition, name)) => (partition.to_string(), name.to_string()),
+ None => (String::new(), file_path.clone()),
+ };
+ let is_log_file = file_name.starts_with('.') && file_name.contains(".log.");
+ let request = requests_by_partition
+ .entry(partition.clone())
+ .or_insert_with(|| HoodieRollbackRequest {
+ partition_path: partition.clone(),
+ file_id: None,
+ latest_base_instant: None,
+ files_to_be_deleted: Vec::new(),
+ log_blocks_to_be_deleted: None,
+ });
+ if is_log_file {
+ request
+ .log_blocks_to_be_deleted
+ .get_or_insert_with(HashMap::new)
+ .insert(file_path.clone(), 1);
+ } else {
+ request.files_to_be_deleted.push(file_path.clone());
+ }
+ let deleted = storage.delete_file(&file_path).await.is_ok();
+ let meta = partition_meta.entry(partition.clone()).or_insert_with(|| {
+ HoodieRollbackPartitionMetadata {
+ partition_path: partition.clone(),
+ ..Default::default()
+ }
+ });
+ if deleted {
+ total_deleted += 1;
+ meta.success_delete_files.push(file_path);
+ } else {
+ meta.failed_delete_files.push(file_path);
Review Comment:
Fixed in 1c2fce80 and 7877f6b0: the rollback fails when a file it must delete survives or a metadata-table partition can't be listed.
##########
crates/core/src/write/rollback.rs:
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Eager marker-based rollback of failed writes (Java `EAGER` cleaning policy).
+//!
+//! At write start, any pending data instant (fenced but never completed) is
+//! rolled back: its marker-listed base/log files are deleted (tv8+ rollback is
+//! deletion-only), its completed-but-orphaned MDT deltacommit and MDT log
+//! files are removed, a `rollback` instant records the operation, and the
+//! failed instant's fencing files and marker dir are cleaned up.
+
+use std::collections::HashMap;
+
+use crate::Result;
+use crate::metadata::rollback::{
+ HoodieInstantInfo, HoodieRollbackMetadata, HoodieRollbackPartitionMetadata, HoodieRollbackPlan,
+ HoodieRollbackRequest,
+};
+use crate::storage::Storage;
+use crate::table::Table;
+use crate::write::append::{generate_instant_time, is_layout_two, timeline_dir};
+use crate::write::markers::{delete_marker_dir, parse_marker_name, read_markers};
+
+const METADATA_BASE: &str = ".hoodie/metadata";
+
+/// A pending (fenced, never completed) instant on the data timeline.
+#[derive(Debug, Clone)]
+struct PendingInstant {
+ timestamp: String,
+ action: String,
+ /// Fencing file names present on the timeline (to delete after rollback).
+ fencing_files: Vec<String>,
+}
+
+/// Roll back all pending data instants (Java `rollbackFailedWrites`, EAGER).
+///
+/// Single-writer: any pending instant belongs to a crashed previous write.
+/// Rollback instants themselves and MDT-internal instants are never targets.
+pub(crate) async fn rollback_failed_writes(table: &Table) -> Result<()> {
+ let storage = table.file_system_view.storage.clone();
+ let timeline = timeline_dir(table);
+ let pending = find_pending_instants(storage.as_ref(), &timeline).await?;
+ for instant in pending {
+ rollback_instant(table, storage.as_ref(), &timeline, &instant).await?;
+ }
+ Ok(())
+}
+
+/// The action infix of a fencing file name: `{ts}.{action}.requested|inflight`
+/// or the bare COW forms `{ts}.inflight` / `{ts}.requested` (empty action).
+///
+/// Must not depend on which of an instant's fencing files is listed first —
+/// directory listing order is filesystem-specific (ext4 returns hash order,
+/// so `.inflight` can precede `.commit.requested`).
+fn action_from_fencing_name(name: &str, timestamp: &str) -> String {
+ let rest = name.trim_start_matches(timestamp).trim_start_matches('.');
+ match rest {
+ "inflight" | "requested" => String::new(),
+ other => other
+ .trim_end_matches(".requested")
+ .trim_end_matches(".inflight")
+ .to_string(),
+ }
+}
+
+async fn find_pending_instants(
+ storage: &Storage,
+ timeline_dir: &str,
+) -> Result<Vec<PendingInstant>> {
+ let files = match storage.list_files(Some(timeline_dir)).await {
+ Ok(files) => files,
+ Err(crate::storage::error::StorageError::ObjectStoreError(
+ object_store::Error::NotFound { .. },
+ )) => return Ok(Vec::new()),
+ Err(error) => return Err(error.into()),
+ };
+ let mut completed = std::collections::HashSet::new();
+ let mut fenced: HashMap<String, (String, Vec<String>)> = HashMap::new();
+ for file in &files {
+ let name = &file.name;
+ let Some(first) = name.chars().next() else {
+ continue;
+ };
+ if !first.is_ascii_digit() {
+ continue;
+ }
+ let timestamp: String = name.chars().take_while(char::is_ascii_digit).collect();
+ if name.ends_with(".requested") || name.ends_with(".inflight") {
+ let action = action_from_fencing_name(name, ×tamp);
+ let entry = fenced
+ .entry(timestamp)
+ .or_insert_with(|| (action.clone(), Vec::new()));
+ // Prefer the action-bearing name (COW inflight has no action infix).
+ if entry.0.is_empty() && !action.is_empty() {
+ entry.0 = action;
+ }
+ entry.1.push(name.clone());
+ } else {
+ completed.insert(timestamp);
+ }
+ }
+ let mut pending: Vec<PendingInstant> = fenced
+ .into_iter()
+ .filter(|(ts, _)| !completed.contains(ts))
+ // Rollback/compaction-style instants are never rolled back here.
+ .filter(|(_, (action, _))| {
+ matches!(
+ action.as_str(),
+ "commit" | "deltacommit" | "replacecommit" | ""
+ )
+ })
+ .map(|(timestamp, (action, fencing_files))| PendingInstant {
+ timestamp,
+ action: if action.is_empty() {
+ "commit".to_string()
+ } else {
+ action
+ },
+ fencing_files,
+ })
+ .collect();
+ pending.sort_by(|a, b| a.timestamp.cmp(&b.timestamp));
+ Ok(pending)
+}
+
+async fn rollback_instant(
+ table: &Table,
+ storage: &Storage,
+ timeline_dir: &str,
+ pending: &PendingInstant,
+) -> Result<()> {
+ let start_time = std::time::Instant::now();
+ let failed_ts = &pending.timestamp;
+
+ // Markers tell us exactly what the failed write intended to put. A crash
+ // before markers means no data or MDT files were written (markers are
+ // written first), so an empty list still produces a valid rollback.
+ let markers = read_markers(storage, failed_ts).await?;
+ let mut requests_by_partition: HashMap<String, HoodieRollbackRequest> = HashMap::new();
+ let mut partition_meta: HashMap<String, HoodieRollbackPartitionMetadata> = HashMap::new();
+ let mut total_deleted = 0i32;
+ for marker in &markers {
+ let Some((file_path, _io_type)) = parse_marker_name(marker) else {
+ continue;
+ };
+ let (partition, file_name) = match file_path.rsplit_once('/') {
+ Some((partition, name)) => (partition.to_string(), name.to_string()),
+ None => (String::new(), file_path.clone()),
+ };
+ let is_log_file = file_name.starts_with('.') && file_name.contains(".log.");
+ let request = requests_by_partition
+ .entry(partition.clone())
+ .or_insert_with(|| HoodieRollbackRequest {
+ partition_path: partition.clone(),
+ file_id: None,
+ latest_base_instant: None,
+ files_to_be_deleted: Vec::new(),
+ log_blocks_to_be_deleted: None,
Review Comment:
Fixed in aad6285a: rollback plans use the version 8 marker-based shape and pending rollbacks resume on their own instant. 4ebb5d43, 563619cd, f33d446c and ccd4adbd harden plan path matching and resume checks.
##########
crates/core/src/write/archival.rs:
##########
@@ -0,0 +1,562 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Timeline archival into the LSM history timeline (Java `TimelineArchiverV2`
+//! + `LSMTimelineWriter`).
+//!
+//! When the active timeline holds more than `hoodie.keep.max.commits`
+//! completed commits, the oldest are archived down to `hoodie.keep.min.commits`:
+//! written as one level-0 parquet of `HoodieLSMTimelineInstant` rows under
+//! `{timeline}/history/`, registered in a new `manifest_{N}` (JSON) pointed at
+//! by `_version_`, then removed from the active timeline (pending fencing
+//! files first, completed files after — Java's ordering). Runs inside the
+//! completion critical section (user design; Java runs it post-commit).
+//!
+//! LSM level compaction (merging L0 files into L1+) is not yet implemented —
+//! readers are unaffected since every file stays manifest-listed.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use arrow_array::{Array, ArrayRef, BinaryArray, Int32Array, RecordBatch, StringArray};
+use arrow_schema::{DataType, Field, Schema};
+use serde::{Deserialize, Serialize};
+
+use crate::Result;
+use crate::error::CoreError;
+use crate::storage::Storage;
+
+/// Java `hoodie.keep.min.commits` default.
+pub(crate) const DEFAULT_KEEP_MIN_COMMITS: usize = 20;
+/// Java `hoodie.keep.max.commits` default.
+pub(crate) const DEFAULT_KEEP_MAX_COMMITS: usize = 30;
+/// Java `hoodie.timeline.manifest.retained.versions` default.
+const RETAINED_MANIFEST_VERSIONS: i64 = 3;
+
+/// Java `HoodieLSMTimelineManifest`: `{"files":[{"fileName":..,"fileLen":..}]}`.
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+struct LsmManifest {
+ files: Vec<LsmFileEntry>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+struct LsmFileEntry {
+ file_name: String,
+ file_len: i64,
+}
+
+#[derive(Debug, Clone)]
+struct CompletedInstant {
+ requested: String,
+ completion: String,
+ action: String,
+ file_name: String,
+}
+
+/// Archive the timeline at `timeline_dir` if it exceeds `max_keep` completed
+/// commits, retaining the newest `min_keep`.
+pub(crate) async fn archive_timeline_if_needed(
+ storage: &Storage,
+ timeline_dir: &str,
+ min_keep: usize,
+ max_keep: usize,
+) -> Result<()> {
+ let listed = match storage.list_files(Some(timeline_dir)).await {
+ Ok(files) => files,
+ Err(crate::storage::error::StorageError::ObjectStoreError(
+ object_store::Error::NotFound { .. },
+ )) => return Ok(()),
+ Err(error) => return Err(error.into()),
+ };
+
+ let mut completed: Vec<CompletedInstant> = Vec::new();
+ let mut fencing_by_instant: HashMap<String, Vec<String>> = HashMap::new();
+ let mut earliest_pending: Option<String> = None;
+ for file in &listed {
+ let name = &file.name;
+ let Some(first) = name.chars().next() else {
+ continue;
+ };
+ if !first.is_ascii_digit() {
+ continue;
+ }
+ let requested: String = name.chars().take_while(char::is_ascii_digit).collect();
+ if name.ends_with(".requested") || name.ends_with(".inflight") {
+ fencing_by_instant
+ .entry(requested.clone())
+ .or_default()
+ .push(name.clone());
+ continue;
+ }
+ // Completed layout-v2 name: `{req}_{comp}.{action}`.
+ let Some((stem, action)) = name.rsplit_once('.') else {
+ continue;
+ };
+ let Some((req, comp)) = stem.split_once('_') else {
+ continue;
+ };
+ completed.push(CompletedInstant {
+ requested: req.to_string(),
+ completion: comp.to_string(),
+ action: action.to_string(),
+ file_name: name.clone(),
+ });
+ let _ = requested;
+ }
+ // Pending = fenced without a completed file.
+ let completed_set: std::collections::HashSet<&str> =
+ completed.iter().map(|c| c.requested.as_str()).collect();
+ for ts in fencing_by_instant.keys() {
+ if !completed_set.contains(ts.as_str()) && earliest_pending.as_ref().is_none_or(|e| ts < e)
+ {
+ earliest_pending = Some(ts.clone());
+ }
+ }
+
+ completed.sort_by(|a, b| a.requested.cmp(&b.requested));
+ // A replacecommit's partitionToReplaceFileIds is what hides the file
+ // groups it replaced; until a cleaner physically deletes those groups,
+ // archiving it resurrects their rows. Java blocks archival at the first
+ // uncleaned replace instant; with no cleaner implemented, that means
+ // never archiving at or past the earliest active replacecommit.
+ let earliest_replace = completed
+ .iter()
+ .filter(|c| c.action == "replacecommit")
+ .map(|c| c.requested.clone())
+ .min();
+ // A savepoint pins its commit and everything after it: archiving past one
+ // would discard the very files the savepoint exists to retain (Java's
+ // TimelineArchiverV2 stops at the first savepoint unless
+ // hoodie.archive.beyond.savepoint, default false).
+ let earliest_savepoint = completed
+ .iter()
+ .filter(|c| c.action == "savepoint")
+ .map(|c| c.requested.clone())
+ .min();
+ let earliest_pending = [earliest_pending, earliest_replace, earliest_savepoint]
+ .into_iter()
+ .flatten()
+ .min();
+ let is_commit_action =
+ |action: &str| matches!(action, "commit" | "deltacommit" | "replacecommit");
+ let commit_count = completed
+ .iter()
+ .filter(|c| is_commit_action(&c.action))
+ .count();
+ // Misconfigured min > max would underflow below and archive the whole
+ // timeline; treat min as the effective floor.
+ let min_keep = min_keep.min(max_keep);
+ if commit_count <= max_keep {
+ return Ok(());
+ }
+
+ // Archive the oldest commits down to `min_keep`, but never at or past the
+ // earliest pending instant; sweep along non-commit actions (rollbacks) up
+ // to the newest archived commit.
+ let mut commits_to_archive = commit_count - min_keep;
+ let mut max_archived_commit: Option<String> = None;
+ let mut to_archive: Vec<CompletedInstant> = Vec::new();
Review Comment:
Fixed in 56fe8cd6, and b4bf0925 stops archival when a fencing file delete fails.
##########
crates/core/src/write/keygen.rs:
##########
@@ -0,0 +1,351 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Derive Hoodie record keys and hive-style partition paths from write batches.
+
+use arrow::array::{Array, StringArray};
+use arrow::record_batch::RecordBatch;
+
+use crate::Result;
+use crate::config::HudiConfigs;
+use crate::config::table::HudiTableConfig::{
+ IsHiveStylePartitioning, PartitionFields, RecordKeyFields,
+};
+use crate::error::CoreError;
+use crate::index::HoodieKey;
+
+/// Build per-row [`HoodieKey`]s from a write batch.
+///
+/// When `hoodie.table.recordkey.fields` is unset/empty, keys are auto-generated
+/// like Java `AutoRecordGenWrapper*KeyGenerator`: `{instant}_{partitionId}_{rowId}`.
+///
+/// `row_id_offset` continues the auto-key row counter across size-split chunks and
+/// partitions within one commit. `partition_id` distinguishes writer partitions
+/// (Java `HoodieRecord.generateSequenceId`).
+///
+/// Partition paths use hive-style (`field=value/...`) when
+/// `hoodie.datasource.write.hive_style_partitioning` is true; otherwise value-only
+/// segments. Null partition values are rejected. Partition columns are not stripped
+/// from the batch — callers must leave them in the written Parquet.
+/// Placeholder instant for pre-fence validation passes: [`hoodie_keys_for_batch`]
+/// uses the instant only to FORMAT auto-generated keys, never to decide errors,
+/// so deriving keys with this dummy value surfaces every deterministic input
+/// error (missing/null/wrong-type key or partition columns) before an instant
+/// is requested.
+pub(crate) const VALIDATION_INSTANT: &str = "00000000000000000";
+
+/// Run key/partition derivation for its error side only — call before fencing
+/// an instant so invalid input leaves no timeline trace.
+pub(crate) fn validate_keygen_inputs(
+ hudi_configs: &HudiConfigs,
+ batches: &[RecordBatch],
+) -> Result<()> {
+ for batch in batches {
+ hoodie_keys_for_batch(hudi_configs, batch, Some(VALIDATION_INSTANT))?;
+ }
+ Ok(())
+}
+
+pub fn hoodie_keys_for_batch(
+ hudi_configs: &HudiConfigs,
+ batch: &RecordBatch,
+ instant: Option<&str>,
+) -> Result<Vec<HoodieKey>> {
+ hoodie_keys_for_batch_with_offset(hudi_configs, batch, instant, 0, 0)
+}
+
+/// Like [`hoodie_keys_for_batch`], with an explicit auto-key row offset and partition id.
+pub fn hoodie_keys_for_batch_with_offset(
+ hudi_configs: &HudiConfigs,
+ batch: &RecordBatch,
+ instant: Option<&str>,
+ row_id_offset: usize,
+ partition_id: u32,
+) -> Result<Vec<HoodieKey>> {
+ let record_key_fields: Option<Vec<String>> =
+ hudi_configs.try_get(RecordKeyFields)?.map(|v| v.into());
+ let auto_keys = record_key_fields
+ .as_ref()
+ .map(|f| f.is_empty() || (f.len() == 1 && f[0].is_empty()))
+ .unwrap_or(true);
+
+ let partition_fields: Vec<String> = hudi_configs.get_or_default(PartitionFields).into();
+ let hive_style: bool = hudi_configs.get_or_default(IsHiveStylePartitioning).into();
+
+ if auto_keys {
+ let instant = instant.ok_or_else(|| {
+ CoreError::Write(
+ "auto record-key generation requires a commit instant timestamp".to_string(),
+ )
+ })?;
+ let mut keys = Vec::with_capacity(batch.num_rows());
+ for row in 0..batch.num_rows() {
+ // Match Java HoodieRecord.generateSequenceId(instant, partitionId, rowId).
+ let record_key = format!("{instant}_{partition_id}_{}", row_id_offset + row);
+ let partition_path = partition_path_for_row(batch, row, &partition_fields, hive_style)?;
+ keys.push(HoodieKey {
+ record_key,
+ partition_path,
+ });
+ }
+ return Ok(keys);
+ }
+
+ let Some(fields) = record_key_fields else {
+ // Unreachable: auto_keys above covers None, but the no-panic rule
+ // wants an error, not an unwrap.
+ return Err(CoreError::Config(
+ crate::config::error::ConfigError::NotFound("record key fields".to_string()),
+ ));
+ };
+ if fields.len() != 1 {
+ return Err(CoreError::Unsupported(
+ "writes currently require exactly one record key field (or none for auto keys)"
+ .to_string(),
+ ));
+ }
+ let key_name = &fields[0];
+ let key_array = batch
+ .column_by_name(key_name)
+ .ok_or_else(|| {
+ CoreError::Schema(format!(
+ "record key field '{key_name}' is missing from write batch"
+ ))
+ })?
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .ok_or_else(|| {
+ CoreError::Unsupported(format!(
+ "record key field '{key_name}' must be Utf8 for writes"
+ ))
+ })?;
+
+ let mut keys = Vec::with_capacity(batch.num_rows());
+ for row in 0..batch.num_rows() {
+ if key_array.is_null(row) {
+ return Err(CoreError::Write(
+ "record key values must be non-null".to_string(),
+ ));
+ }
+ let record_key = key_array.value(row).to_string();
+ let partition_path = partition_path_for_row(batch, row, &partition_fields, hive_style)?;
+ keys.push(HoodieKey {
+ record_key,
+ partition_path,
+ });
+ }
+ Ok(keys)
+}
+
+fn partition_path_for_row(
Review Comment:
Fixed in 60d96990: write keys honor the table key generator, url-encoded partitions, and the default partition. bda5901f, 91ef9912, a14c7ced and 60b768a2 cover complex-keygen validation, guard placement, key-filter deletes, and partition SET.
##########
crates/core/src/write/rewrite.rs:
##########
@@ -0,0 +1,3924 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+use std::collections::{HashMap, HashSet};
+use std::str::FromStr;
+
+use arrow::array::{Array, ArrayRef, BooleanArray, StringArray, UInt32Array};
+use arrow::compute::concat_batches;
+use arrow::record_batch::RecordBatch;
+use arrow_select::take::take;
+use arrow_select::zip::zip;
+
+use crate::Result;
+use crate::config::table::HudiTableConfig::{OrderingFields, RecordKeyFields, RecordMergeStrategy};
+use crate::error::CoreError;
+use crate::expr::ExprOperator;
+use crate::expr::filter::{Filter, filters_to_row_mask, validate_fields_against_schemas};
+use crate::file_group::record_batches::RecordBatches;
+use crate::index::{HoodieIndex, HoodieKey, is_record_index_enabled};
+use crate::merge::RecordMergeStrategyValue;
+use crate::merge::record_merger::RecordMerger;
+use crate::metadata::commit::{HoodieCommitMetadata, HoodieWriteStat};
+use crate::metadata::meta_field::MetaField;
+use crate::metadata::table::encode::RecordIndexEntry;
+use crate::table::{ReadOptions, Table};
+use crate::timeline::instant::{Action, Instant};
+use crate::write::append::{
+ ensure_copy_on_write, generate_instant_time, is_layout_two, prepare_batches_for_write,
+ timeline_dir,
+};
+use crate::write::keygen::{hoodie_keys_for_batch, relative_data_path};
+use crate::write::metadata::{
+ instant_to_epoch_millis, is_column_stats_enabled, update_column_stats_partitions,
+ update_files_partition_entries, update_record_index,
+};
+
+/// Result of an upsert, delete, or overwrite write.
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub struct WriteResult {
+ /// Completed instant timestamp.
+ pub instant: String,
+ /// Number of rows written to replacement base files.
+ pub num_writes: usize,
+ /// Number of input keys that replaced existing rows.
+ pub num_updates: usize,
+ /// Number of input keys not previously present.
+ pub num_inserts: usize,
+ /// Number of rows deleted.
+ pub num_deletes: usize,
+}
+
+/// Options controlling an upsert.
+#[derive(Clone, Debug, Default, PartialEq, Eq)]
+pub struct UpsertOptions {
+ /// Data columns to overwrite for matched rows. `None` replaces complete rows.
+ pub update_columns: Option<Vec<String>>,
+}
+
+/// How a COW rewrite commits: same file-group slice vs replacecommit overwrite.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+enum RewriteKind {
+ /// New base file slice in the same file group(s); timeline action = commit.
+ Commit,
+ /// Replace file groups; timeline action = replacecommit (overwrite only for now).
+ Replace,
+}
+
+/// Critical section 1: request the action — mint the instant time and fence
+/// it (requested + inflight) under the table lock, then release; the write's
+/// heavy work runs unlocked. `request_commit` / `request_replacecommit` pick
+/// the timeline action from the rewrite kind.
+async fn request_rewrite_instant(
+ table: &Table,
+ operation: &str,
+ kind: RewriteKind,
+) -> Result<String> {
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ let _cs1 = lock.lock().await?;
+ let instant = generate_instant_time().await;
+ let action = match kind {
+ RewriteKind::Commit => Action::Commit,
+ RewriteKind::Replace => Action::ReplaceCommit,
+ };
+ let requested_bytes = if kind == RewriteKind::Replace {
+ crate::metadata::replace_commit::HoodieRequestedReplaceMetadata::for_operation(operation)
+ .to_avro_bytes()?
+ } else {
+ Vec::new()
+ };
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir(table),
+ &instant,
+ action,
+ requested_bytes,
+ crate::write::inflight_commit_metadata_bytes(operation, is_layout_two(table))?,
+ )
+ .await?;
+ Ok(instant)
+}
+
+/// Critical section 1 for MOR deltacommits.
+async fn request_deltacommit(table: &Table, operation: &str) -> Result<String> {
+ let storage = table.file_system_view.storage.clone();
+ let lock = crate::write::lock::lock_provider_for(table);
+ let _cs1 = lock.lock().await?;
+ let instant = generate_instant_time().await;
+ crate::write::fence_timeline_instant(
+ storage.as_ref(),
+ &timeline_dir(table),
+ &instant,
+ Action::DeltaCommit,
+ Vec::new(),
+ crate::write::inflight_commit_metadata_bytes(operation, is_layout_two(table))?,
+ )
+ .await?;
+ Ok(instant)
+}
+
+pub async fn upsert_batches(
+ table: &mut Table,
+ batches: &[RecordBatch],
+ options: UpsertOptions,
+) -> Result<WriteResult> {
+ table.reload_timeline_for_write().await?;
+ ensure_configured_record_key(table)?;
+ if table.is_mor() {
+ return mor_upsert_batches(table, batches, options).await;
+ }
+ ensure_rewrite_supported(table)?;
+ if batches.is_empty() {
+ return Err(CoreError::Write(
+ "upsert requires at least one RecordBatch".to_string(),
+ ));
+ }
+ validate_write_input(table, batches, "upsert").await?;
+ if let Some(columns) = &options.update_columns {
+ let schema = batches[0].schema();
+ for column in columns {
+ if schema.field_with_name(column).is_err() {
+ return Err(CoreError::Schema(format!(
+ "update column '{column}' is not in the write batch schema"
+ )));
+ }
+ if MetaField::field_names_with_operation().contains(&column.as_str()) {
+ return Err(CoreError::Write(format!(
+ "update column '{column}' is a Hudi meta field"
+ )));
+ }
+ }
+ }
+ crate::write::keygen::validate_keygen_inputs(&table.hudi_configs, batches)?;
+ let instant = request_rewrite_instant(table, "UPSERT", RewriteKind::Commit).await?;
+ let incoming = prepare_batches_for_write(&table.hudi_configs, batches, &instant, "pending")?;
+ let incoming = concat(&incoming)?;
+ let key_name = record_key_name(table, &incoming)?;
+ let incoming = deduplicate_last_by_key(&incoming, &key_name, &table.hudi_configs)?;
+ let incoming_hoodie_keys = hoodie_keys_for_batch(&table.hudi_configs, &incoming, None)?;
+ let locations = crate::index::for_table_checked(table)
+ .await
+ .tag_location(table, &incoming_hoodie_keys)
+ .await?;
+ // Small-file packing (Java UpsertPartitioner.assignInserts): assign insert
+ // rows to existing small file groups first; chosen groups join the read
+ // set so their rows are preserved through the rewrite.
+ let sizing = crate::write::sizing::SizingConfig::from_table(table);
+ let avg_record_size = crate::write::sizing::average_record_size(
+ table,
+ table.file_system_view.storage.as_ref(),
+ &sizing,
+ )
+ .await;
+ let slices_by_partition = crate::write::sizing::latest_slices_by_partition(table).await?;
+ let mut insert_rows_by_partition: HashMap<String, Vec<usize>> = HashMap::new();
+ for (index, hoodie_key) in incoming_hoodie_keys.iter().enumerate() {
+ if locations.get(hoodie_key).and_then(Option::as_ref).is_none() {
+ insert_rows_by_partition
+ .entry(hoodie_key.partition_path.clone())
+ .or_default()
+ .push(index);
+ }
+ }
+ let mut insert_target_by_row: HashMap<usize, Option<String>> = HashMap::new();
+ for (partition, rows) in &insert_rows_by_partition {
+ let slices: Vec<&crate::file_group::file_slice::FileSlice> = slices_by_partition
+ .get(partition)
+ .map(|v| v.iter().collect())
+ .unwrap_or_default();
+ let small = crate::write::sizing::small_file_groups(&slices, false, &sizing);
+ let assignments = crate::write::sizing::assign_insert_buckets(
+ rows.len(),
+ &small,
+ avg_record_size,
+ &sizing,
+ );
+ let mut cursor = 0usize;
+ for (bucket, count) in assignments {
+ for row in &rows[cursor..cursor + count] {
+ match &bucket {
+ crate::write::sizing::InsertBucket::Existing { file_id } => {
+ insert_target_by_row.insert(*row, Some(file_id.clone()));
+ }
+ crate::write::sizing::InsertBucket::New => {
+ insert_target_by_row.insert(*row, None);
+ }
+ }
+ }
+ cursor += count;
+ }
+ }
+ // MergeHandle model: route each input row to its target file group, then
+ // per-group workers read+merge+write. The driver only holds the input and
+ // plan metadata.
+ let mut slice_by_file_id: HashMap<String, (String, crate::file_group::file_slice::FileSlice)> =
+ HashMap::new();
+ for (partition, slices) in &slices_by_partition {
+ for slice in slices {
+ slice_by_file_id.insert(
+ slice.file_id().to_string(),
+ (partition.clone(), slice.clone()),
+ );
+ }
+ }
+
+ let mut updates = 0usize;
+ let mut inserts = 0usize;
+ let mut rows_by_group: HashMap<String, Vec<u32>> = HashMap::new();
+ let mut new_rows_by_partition: HashMap<String, Vec<u32>> = HashMap::new();
+ let mut row_target: Vec<Option<String>> = Vec::with_capacity(incoming.num_rows());
+ // Update-partition-path (global index): a key upserted with a different
+ // partition value MOVES — evicted from the group in its old partition and
+ // inserted where its values say. Anything else leaves rows whose stored
+ // partition columns disagree with the path they physically live in.
+ let mut evictions_by_group: HashMap<String, HashSet<String>> = HashMap::new();
+ // Merge-before-move (Java parity): find the cross-partition updates whose
+ // incoming row loses the event-time merge; they stay in their old group.
+ let mut moved_candidates: HashMap<String, Vec<String>> = HashMap::new();
Review Comment:
Kept as is and documented in 5c966d3f: upserts move a record whose partition changes, unlike Java's default `hoodie.record.index.update.partition.path=false`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4409410071
##########
crates/core/src/config/table.rs:
##########
@@ -145,6 +145,12 @@ pub enum HudiTableConfig {
/// When creating a metadata table instance, this value should be passed as the
/// PartitionFields option.
MetadataTablePartitions,
+
+ /// Enable the metadata-table record-level index (RLI) used to tag upsert/delete keys.
+ ///
+ /// When false, writers fall back to [`crate::index::SimpleIndex`] (full table scan).
+ /// Requires [`Self::MetadataTableEnabled`].
+ RecordIndexEnabled,
Review Comment:
Addressed in #851 (2c10dffa): the independent `RecordIndexEnabled` config is gone, and whether the record index is on is read from `hoodie.table.metadata.partitions`.
##########
crates/core/src/config/table.rs:
##########
@@ -175,6 +181,7 @@ impl AsRef<str> for HudiTableConfig {
Self::TimelineHistoryPath => "hoodie.timeline.history.path",
Self::MetadataTableEnabled => "hoodie.metadata.enable",
Self::MetadataTablePartitions => "hoodie.table.metadata.partitions",
+ Self::RecordIndexEnabled => "hoodie.metadata.record.index.enable",
Review Comment:
Addressed in #851 (2c10dffa): the old key went away with `RecordIndexEnabled`, and the table builder rejects `hoodie.metadata.global.record.level.index.enable` through `with_option`, since `with_record_index` sets it (8420bab1).
##########
crates/core/src/metadata/commit.rs:
##########
@@ -193,21 +208,6 @@ impl HoodieCommitMetadata {
.unwrap_or_default()
}
- /// Get the file IDs to be replaced for a specific partition
- pub fn get_partition_replace_file_ids(&self, partition: &str) -> Option<&Vec<String>> {
Review Comment:
Done in #851 (48736922): `partition_to_replace_file_ids` and its accessors are back on `HoodieCommitMetadata`, with their tests.
##########
crates/core/src/metadata/table/records.rs:
##########
@@ -282,7 +282,11 @@ pub fn decode_files_partition_record_with_schema(
});
}
- let avro_value = decode_avro_value(value, schema)?;
+ // Prefer the record's own writer schema (from the HFile or log block it
+ // was read from) — `schema` is only a fallback for records with no
+ // container schema. Mixed-writer tables (rs + Spark) have differing
+ // HoodieMetadataRecord schemas per container.
Review Comment:
Addressed in #851 (bdacbd6a): `HFileRecord` is a plain key/value pair again and the comment is gone. Each HFile or log block now hands its writer schema to the decoder with its records.
##########
crates/core/src/lib.rs:
##########
@@ -46,12 +46,17 @@
//! }
//! ```
+// The write-path async fns (e.g. `Table::delete_keys`) nest deeply enough that computing their
+// layout exceeds the default limit of 128 on Linux targets.
+#![recursion_limit = "256"]
+
Review Comment:
Tracked in #860, keeping the limit for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4409410367
##########
crates/core/src/config/table.rs:
##########
@@ -175,6 +181,7 @@ impl AsRef<str> for HudiTableConfig {
Self::TimelineHistoryPath => "hoodie.timeline.history.path",
Self::MetadataTableEnabled => "hoodie.metadata.enable",
Self::MetadataTablePartitions => "hoodie.table.metadata.partitions",
+ Self::RecordIndexEnabled => "hoodie.metadata.record.index.enable",
Review Comment:
Addressed in #851 (2c10dffa): the old key went away with `RecordIndexEnabled`, and since `with_record_index` sets `hoodie.metadata.global.record.level.index.enable`, the builder now rejects it through `with_option`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): native table write support with spark storage parity [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #666:
URL: https://github.com/apache/hudi-rs/pull/666#discussion_r4409410367
##########
crates/core/src/config/table.rs:
##########
@@ -175,6 +181,7 @@ impl AsRef<str> for HudiTableConfig {
Self::TimelineHistoryPath => "hoodie.timeline.history.path",
Self::MetadataTableEnabled => "hoodie.metadata.enable",
Self::MetadataTablePartitions => "hoodie.table.metadata.partitions",
+ Self::RecordIndexEnabled => "hoodie.metadata.record.index.enable",
Review Comment:
Addressed in #851: the old key went away with `RecordIndexEnabled` (2c10dffa), and `with_option` now reads `hoodie.metadata.global.record.level.index.enable`, or its alternative `hoodie.metadata.record.index.enable`, as `with_record_index`, with the primary key taking precedence like the Java write config (7d9382b5).
##########
crates/core/src/storage/mod.rs:
##########
@@ -398,6 +400,78 @@ impl Storage {
}
Ok(file_metadata)
}
+
+ /// Resolve a relative path to an [`ObjPath`] under this storage's base URL.
+ fn relative_obj_path(&self, relative_path: &str) -> Result<ObjPath> {
+ let obj_url = join_url_segments(&self.base_url, &[relative_path])?;
+ Ok(ObjPath::from_url_path(obj_url.path())?)
+ }
+
+ /// Write bytes to a path relative to the table base path.
+ pub async fn put_file(&self, relative_path: &str, bytes: impl Into<Bytes>) -> Result<()> {
+ let obj_path = self.relative_obj_path(relative_path)?;
+ self.object_store
+ .put(&obj_path, PutPayload::from(bytes.into()))
+ .await?;
+ Ok(())
+ }
+
+ /// Write a file only if nothing exists at the path (`PutMode::Create`).
+ ///
+ /// Timeline files must never silently replace one another: two writers
+ /// minting the same instant otherwise clobber each other's commit with no
+ /// error. An existing object surfaces as
+ /// [`StorageError::AlreadyExists`](error::StorageError) so callers can
+ /// report the conflict.
+ pub async fn put_file_if_absent(
Review Comment:
It does already: `put_file_if_absent` uses `PutMode::Create`, and object_store 0.13 sends that to S3 as a conditional PUT (`If-None-Match: *`, its default `ETagMatch` mode). The create is not retried on a timeout, so a retry cannot turn a PUT that landed into `AlreadyExists`. Setting `aws_conditional_put=disabled` makes these writes fail with `NotImplemented` rather than overwrite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org