You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by vh...@apache.org on 2026/08/05 07:29:28 UTC

(hudi) branch release-1.2.1 created (now dbfafbcf5738)

This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a change to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git


      at dbfafbcf5738 feat(spark): add restore_to_instant stored procedure (#18696)

This branch includes the following new commits:

     new 9faeca0351bc chore(docker): reduce base_java17 and spark_base image size (#18542)
     new d030eb17dd89 test(spark): Add date logical type test to TestAvroConversionUtils (#18584)
     new 69ec50cc07c4 feat(common): roll over commit metadata to clean (#18590)
     new 481eb8047c65 refactor: move checkpoint metadata lookup helper to hudi-common (#18489)
     new b5342e756b83 feat(flink): Backport Flink 2.1 nested Parquet column readers and INT64 timestamp dispatch (FLINK-35702) (#18636)
     new 08584dabc8c4 feat(flink): Wire Flink 2.1 nested Parquet readers into the Hudi read path (FLINK-35702) (#18700)
     new f0bf5eb7647d refactor(flink): Remove legacy Parquet nested readers superseded by Flink 2.1 Dremel path (FLINK-35702) (#18701)
     new 276565468ef0 docs: Document muttley package as internal/optional for OSS users (#18394)
     new b85f1cc3e14a perf: Reduce unnecessary FSDataOutputStream#hsync to enhance append performance (#17517)
     new 062e4c6ca578 fix(flink): add Apache license header to muttley/README.md (#18713)
     new 53559242345b feat: add variant type adapter for Flink (#18702)
     new a3272bd3293a chore: migrate the flink ITs run to flink2.1 (#18717)
     new 64974eb0e9e9 feat(utilities): add Spark/HoodieStreamer validators for pre-commit validation - Phase 3 (#18405)
     new 9023d94478d5 [MINOR] Fix typos in comments and assertion messages (#18763)
     new 41717512f0c0 fix(flink): enforce Parquet VARIANT annotation in Flink schema conversion for unshredded variant (#18539)
     new dbfafbcf5738 feat(spark): add restore_to_instant stored procedure (#18696)

The 16 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



(hudi) 15/16: fix(flink): enforce Parquet VARIANT annotation in Flink schema conversion for unshredded variant (#18539)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 41717512f0c00ec5dd1d5a2c8b91210f244220a5
Author: Krishen <22...@users.noreply.github.com>
AuthorDate: Mon May 18 21:26:41 2026 -0700

    fix(flink): enforce Parquet VARIANT annotation in Flink schema conversion for unshredded variant (#18539)
    
    * feat(flink): write/read unshredded variant to Flink parquet file writers/readers using Flink's Variant type
    
    ---------
    
    Co-authored-by: Krishen Bhan <β€œbkrishen@uber.com”>
    Co-authored-by: Cursor <cu...@cursor.com>
    (cherry picked from commit 71aa121035cf912770f6ad4bf157a945ae3a8409)
---
 .../row/parquet/ParquetSchemaConverter.java        |  31 ++++-
 .../apache/hudi/util/HoodieSchemaConverter.java    |   4 +
 .../row/parquet/TestParquetSchemaConverter.java    | 148 +++++++++++++++++++++
 .../hudi/util/TestHoodieSchemaConverter.java       |  34 ++++-
 .../ITTestVariantCrossEngineCompatibility.java     |   3 +-
 .../org/apache/hudi/adapter/DataTypeAdapter.java   |  20 ++-
 .../org/apache/hudi/adapter/DataTypeAdapter.java   |  20 ++-
 .../org/apache/hudi/adapter/DataTypeAdapter.java   |  20 ++-
 .../org/apache/hudi/adapter/DataTypeAdapter.java   |  20 ++-
 .../org/apache/hudi/adapter/DataTypeAdapter.java   |  20 ++-
 .../org/apache/hudi/adapter/DataTypeAdapter.java   |  35 +++++
 11 files changed, 323 insertions(+), 32 deletions(-)

diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetSchemaConverter.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetSchemaConverter.java
index 668098314d48..dd19100f8ccd 100644
--- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetSchemaConverter.java
+++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetSchemaConverter.java
@@ -49,6 +49,17 @@ import static org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
 /**
  * Schema converter converts Parquet schema to and from Flink internal types.
  *
+ * <p>On reads, this converter performs best-effort physical type mapping. It detects the
+ * Parquet {@code VARIANT} annotation and will reject shredded variants. Blob and Vector types
+ * cannot be distinguished from ordinary binary columns via Parquet schema alone.
+ *
+ * <p>On writes, this converter maps Flink {@code VariantType} to the canonical unshredded Parquet
+ * layout (group with binary metadata + value fields). The VARIANT logical type annotation is
+ * resolved by {@link DataTypeAdapter#variantParquetAnnotation()} β€” on Flink 2.1+ with
+ * parquet-java 1.16.0+ the annotation is attached automatically; on pre-2.1 Flink or with
+ * parquet < 1.16.0 the write throws {@link UnsupportedOperationException} because writing
+ * variant data without the annotation would produce files that no reader can identify as variant.
+ *
  * <p>Reference org.apache.flink.formats.parquet.utils.ParquetSchemaConverter to support timestamp of INT64 8 bytes.
  */
 @Slf4j
@@ -158,6 +169,9 @@ public class ParquetSchemaConverter {
                 convertToRowField(keyValueType.getLeft()).getType().copy(true),
                 convertToRowField(keyValueType.getRight()).getType()));
       } else if (hasVariantAnnotation(logicalType)) {
+        // Fires for files written with parquet-java that carry the VARIANT annotation.
+        // The reader infers the Flink RowType from the Parquet footer via convertToRowType(),
+        // so this annotation detection is the primary mechanism for recognizing Variant columns.
         if (isShreddedVariant(groupType)) {
           throw new UnsupportedOperationException(
               "Shredded Variant is not supported in Flink. "
@@ -223,10 +237,25 @@ public class ParquetSchemaConverter {
   /**
    * Converts a Variant column to the canonical unshredded Parquet layout:
    * a group with required binary {@code metadata} and required binary {@code value}.
+   *
+   * <p>No shredded-variant guard is needed here: Flink 2.1's {@code VariantType} is a single
+   * atomic {@code LogicalTypeRoot.VARIANT} with no shredding representation (FLIP-521 scopes
+   * shredding out), so a shredded variant can never arrive as a Flink LogicalType.
+   *
+   * <p>Delegates to {@link DataTypeAdapter#variantParquetAnnotation()} for the VARIANT logical
+   * type annotation. On Flink < 2.1 this throws (variant writes are unsupported). On Flink 2.1+
+   * with parquet-java < 1.16.0 this also throws, because writing variant data without the
+   * annotation would produce files that no reader can identify as variant.
    */
   private static Type convertVariantToParquetType(String name, Type.Repetition repetition) {
-    // TODO: add .as(LogicalTypeAnnotation.variantType()) once parquet-java is bumped to 1.16.0
+    LogicalTypeAnnotation annotation = DataTypeAdapter.variantParquetAnnotation()
+        .orElseThrow(() -> new UnsupportedOperationException(
+            "Cannot write Variant columns: parquet-java 1.16.0+ is required to emit the VARIANT "
+                + "logical type annotation. Without the annotation, readers cannot identify the "
+                + "column as Variant. Current parquet-java version does not support "
+                + "LogicalTypeAnnotation.variantType()."));
     return Types.buildGroup(repetition)
+        .as(annotation)
         .addField(Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY, Type.Repetition.REQUIRED)
             .named(HoodieSchema.Variant.VARIANT_METADATA_FIELD))
         .addField(Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY, Type.Repetition.REQUIRED)
diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/HoodieSchemaConverter.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/HoodieSchemaConverter.java
index c49142800b45..c413101dde46 100644
--- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/HoodieSchemaConverter.java
+++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/HoodieSchemaConverter.java
@@ -70,6 +70,10 @@ public class HoodieSchemaConverter {
    * <p>The "{rowName}." is used as the nested row type name prefix in order to generate
    * the right schema. Nested record types that only differ by type name are still compatible.
    *
+   * <p>On Flink 2.1+, {@code LogicalTypeRoot.VARIANT} is detected via string comparison
+   * (to avoid compile-time dependency) and mapped to {@link HoodieSchema#createVariant()}.
+   * Pre-2.1 Flink does not support Variant.
+   *
    * @param logicalType Flink logical type
    * @param rowName     the record name
    * @return HoodieSchema matching this logical type
diff --git a/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/io/storage/row/parquet/TestParquetSchemaConverter.java b/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/io/storage/row/parquet/TestParquetSchemaConverter.java
index 318f2034d175..403b84068496 100644
--- a/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/io/storage/row/parquet/TestParquetSchemaConverter.java
+++ b/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/io/storage/row/parquet/TestParquetSchemaConverter.java
@@ -18,6 +18,10 @@
 
 package org.apache.hudi.io.storage.row.parquet;
 
+import org.apache.hudi.adapter.DataTypeAdapter;
+import org.apache.hudi.common.schema.HoodieSchema;
+import org.apache.hudi.common.util.Option;
+
 import org.apache.flink.table.api.DataTypes;
 import org.apache.flink.table.types.DataType;
 import org.apache.flink.table.types.logical.ArrayType;
@@ -27,13 +31,19 @@ import org.apache.flink.table.types.logical.DecimalType;
 import org.apache.flink.table.types.logical.DoubleType;
 import org.apache.flink.table.types.logical.FloatType;
 import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.table.types.logical.MapType;
 import org.apache.flink.table.types.logical.RowType;
 import org.apache.flink.table.types.logical.SmallIntType;
 import org.apache.flink.table.types.logical.TimestampType;
 import org.apache.flink.table.types.logical.TinyIntType;
 import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
 import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.apache.parquet.schema.Types;
 import org.junit.jupiter.api.Test;
 
 import java.util.Arrays;
@@ -41,6 +51,9 @@ import java.util.Collections;
 
 import static org.hamcrest.CoreMatchers.is;
 import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
 
 /**
  * Test cases for {@link ParquetSchemaConverter}.
@@ -216,4 +229,139 @@ public class TestParquetSchemaConverter {
         + "}\n";
     assertThat(messageType.toString(), is(expected));
   }
+
+  /**
+   * A Parquet group with metadata + value binary fields but NO VARIANT annotation must be
+   * treated as a plain ROW. Only the Parquet {@code VARIANT} annotation triggers variant
+   * detection in this converter; unannotated groups are never guessed as variant.
+   */
+  @Test
+  void testVariantPhysicalLayoutTreatedAsRow() {
+    MessageType variantParquet = new MessageType(
+        "test",
+        Types.primitive(PrimitiveType.PrimitiveTypeName.INT32,
+            Type.Repetition.REQUIRED).named("id"),
+        Types.buildGroup(Type.Repetition.REQUIRED)
+            .addField(Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY,
+                Type.Repetition.REQUIRED).named("metadata"))
+            .addField(Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY,
+                Type.Repetition.REQUIRED).named("value"))
+            .named("data"));
+
+    RowType rowType = ParquetSchemaConverter.convertToRowType(variantParquet);
+    assertEquals(2, rowType.getFieldCount());
+    assertEquals("ROW", rowType.getTypeAt(1).getTypeRoot().name());
+  }
+
+  /**
+   * Unannotated group with metadata + value + typed_value (3 fields) is treated as a generic
+   * ROW when no annotation or schema hint is present.
+   */
+  @Test
+  void testUnannotatedShreddedGroupTreatedAsRow() {
+    MessageType shreddedNoAnnotation = new MessageType(
+        "test",
+        Types.primitive(PrimitiveType.PrimitiveTypeName.INT32,
+            Type.Repetition.REQUIRED).named("id"),
+        Types.buildGroup(Type.Repetition.REQUIRED)
+            .addField(Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY,
+                Type.Repetition.REQUIRED).named("metadata"))
+            .addField(Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY,
+                Type.Repetition.REQUIRED).named("value"))
+            .addField(Types.primitive(PrimitiveType.PrimitiveTypeName.INT32,
+                Type.Repetition.OPTIONAL).named("typed_value"))
+            .named("data"));
+
+    RowType rowType = ParquetSchemaConverter.convertToRowType(shreddedNoAnnotation);
+    assertEquals(2, rowType.getFieldCount());
+    assertEquals("ROW", rowType.getTypeAt(1).getTypeRoot().name());
+  }
+
+  /**
+   * On Flink 2.1+ with parquet 1.16.0+, converting a RowType containing a Variant column to a
+   * Parquet MessageType should produce a group with the VARIANT annotation and required binary
+   * {@code metadata} and {@code value} fields.
+   * On pre-2.1 Flink this test is skipped since VariantType does not exist.
+   * On parquet < 1.16.0 the write is expected to fail (annotation unavailable).
+   */
+  @Test
+  void testVariantWritePathProducesCorrectLayout() {
+    LogicalType variantType;
+    try {
+      variantType = DataTypeAdapter.createVariantType().getLogicalType();
+    } catch (UnsupportedOperationException e) {
+      // Pre-2.1 Flink: VariantType doesn't exist, skip
+      return;
+    }
+
+    RowType rowType = RowType.of(
+        new LogicalType[]{new IntType(), variantType},
+        new String[]{"id", "data"});
+
+    if (!DataTypeAdapter.variantParquetAnnotation().isPresent()) {
+      // parquet < 1.16.0: write must fail because annotation is unavailable
+      UnsupportedOperationException ex = org.junit.jupiter.api.Assertions.assertThrows(
+          UnsupportedOperationException.class,
+          () -> ParquetSchemaConverter.convertToParquetMessageType("test", rowType));
+      assertTrue(ex.getMessage().contains("parquet-java 1.16.0+"),
+          "Error message should mention parquet version requirement");
+      return;
+    }
+
+    // parquet 1.16.0+: write succeeds with annotation
+    MessageType messageType = ParquetSchemaConverter.convertToParquetMessageType("test", rowType);
+    assertEquals(2, messageType.getFieldCount());
+
+    Type variantField = messageType.getType("data");
+    assertTrue(variantField instanceof GroupType, "Variant column should be a Parquet group");
+    GroupType variantGroup = (GroupType) variantField;
+    assertEquals(2, variantGroup.getFieldCount());
+    assertEquals(HoodieSchema.Variant.VARIANT_METADATA_FIELD, variantGroup.getType(0).getName());
+    assertEquals(HoodieSchema.Variant.VARIANT_VALUE_FIELD, variantGroup.getType(1).getName());
+    assertTrue(variantGroup.getType(0).isPrimitive());
+    assertTrue(variantGroup.getType(1).isPrimitive());
+    assertEquals(PrimitiveType.PrimitiveTypeName.BINARY,
+        variantGroup.getType(0).asPrimitiveType().getPrimitiveTypeName());
+    assertEquals(PrimitiveType.PrimitiveTypeName.BINARY,
+        variantGroup.getType(1).asPrimitiveType().getPrimitiveTypeName());
+    assertNotNull(variantGroup.getLogicalTypeAnnotation(),
+        "Variant group must carry the VARIANT annotation");
+  }
+
+  /**
+   * Verifies that writing a Variant column fails with a clear error when parquet-java on the
+   * classpath does not support the VARIANT annotation (< 1.16.0). On pre-2.1 Flink the adapter
+   * throws directly; on Flink 2.1+ with parquet < 1.16.0 the write path throws.
+   */
+  @Test
+  void testVariantWriteFailsWithoutAnnotation() {
+    Option<LogicalTypeAnnotation> annotationOpt;
+    try {
+      annotationOpt = DataTypeAdapter.variantParquetAnnotation();
+    } catch (UnsupportedOperationException e) {
+      // Pre-2.1 Flink: expected to throw from the adapter
+      assertTrue(e.getMessage().contains("VARIANT type is only supported in Flink 2.1+"));
+      return;
+    }
+
+    if (annotationOpt.isPresent()) {
+      // parquet 1.16.0+: annotation is available, write succeeds β€” nothing to test here
+      return;
+    }
+
+    // Flink 2.1 + parquet < 1.16.0: annotation is null, write must fail
+    LogicalType variantType = DataTypeAdapter.createVariantType().getLogicalType();
+    RowType rowType = RowType.of(
+        new LogicalType[]{new IntType(), variantType},
+        new String[]{"id", "data"});
+
+    UnsupportedOperationException ex = org.junit.jupiter.api.Assertions.assertThrows(
+        UnsupportedOperationException.class,
+        () -> ParquetSchemaConverter.convertToParquetMessageType("test", rowType));
+    assertTrue(ex.getMessage().contains("parquet-java 1.16.0+"),
+        "Error message should mention the parquet version requirement");
+    assertTrue(ex.getMessage().contains("VARIANT"),
+        "Error message should mention VARIANT");
+  }
+
 }
diff --git a/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java b/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java
index 9a4f920b859e..9f6ad8f53bf7 100644
--- a/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java
+++ b/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java
@@ -695,19 +695,16 @@ public class TestHoodieSchemaConverter {
 
   @Test
   public void testVariantTypeConversion() {
-    // Test direct Variant conversion
     HoodieSchema variantSchema = HoodieSchema.createVariant();
     DataType dataType = HoodieSchemaConverter.convertToDataType(variantSchema);
     assertNotNull(dataType);
 
-    // Verify it's a Variant
     assertThat("the return type should be variant",
         dataType.getLogicalType().asSummaryString(), is("VARIANT NOT NULL"));
   }
 
   @Test
   public void testVariantInRecordConversion() {
-    // Test Variant field within a record
     HoodieSchema recordWithVariant = HoodieSchema.createRecord(
         "test_record",
         null,
@@ -722,11 +719,40 @@ public class TestHoodieSchemaConverter {
     assertEquals(2, result.getFieldCount());
     assertEquals("data", result.getFieldNames().get(1));
 
-    // Verify variant field
     assertThat("the return type should be variant",
         result.getTypeAt(1).asSummaryString(), is("VARIANT NOT NULL"));
   }
 
+  @Test
+  public void testVariantInArrayConversion() {
+    HoodieSchema arrayOfVariant = HoodieSchema.createArray(HoodieSchema.createVariant());
+    DataType dataType = HoodieSchemaConverter.convertToDataType(arrayOfVariant);
+    assertNotNull(dataType);
+    assertInstanceOf(ArrayType.class, dataType.getLogicalType());
+    LogicalType elementType = ((ArrayType) dataType.getLogicalType()).getElementType();
+    assertEquals("VARIANT", elementType.getTypeRoot().name());
+  }
+
+  @Test
+  public void testVariantInMapConversion() {
+    HoodieSchema mapOfVariant = HoodieSchema.createMap(HoodieSchema.createVariant());
+    DataType dataType = HoodieSchemaConverter.convertToDataType(mapOfVariant);
+    assertNotNull(dataType);
+    assertInstanceOf(MapType.class, dataType.getLogicalType());
+    LogicalType valueType = ((MapType) dataType.getLogicalType()).getValueType();
+    assertEquals("VARIANT", valueType.getTypeRoot().name());
+  }
+
+  @Test
+  public void testShreddedVariantConversionThrows() {
+    HoodieSchema.Variant shredded = HoodieSchema.createVariantShredded(
+        HoodieSchema.create(HoodieSchemaType.STRING));
+    UnsupportedOperationException ex = assertThrows(
+        UnsupportedOperationException.class,
+        () -> HoodieSchemaConverter.convertToDataType(shredded));
+    assertTrue(ex.getMessage().contains("Shredded Variant is not yet supported in Flink"));
+  }
+
   @Test
   public void testBlobStructureValidation() {
     // Positive case: Create ROW matching BLOB structure
diff --git a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java
index de6e3835d120..e7f9ed3766f1 100644
--- a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java
+++ b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java
@@ -41,6 +41,7 @@ import static org.junit.jupiter.api.Assertions.assertArrayEquals;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertNotNull;
 
+
 /**
  * Integration test for cross-engine compatibility - verifying that Flink can read Variant tables written by Spark 4.0.
  */
@@ -56,7 +57,6 @@ public class ITTestVariantCrossEngineCompatibility {
   private void verifyFlinkCanReadSparkVariantTable(String tablePath, String tableType, String testDescription) throws Exception {
     TableEnvironment tableEnv = TestTableEnvs.getBatchTableEnv();
 
-    // Create a Hudi table pointing to the Spark-written data
     String createTableDdl = String.format(
         "CREATE TABLE variant_table ("
             + "  id INT,"
@@ -73,7 +73,6 @@ public class ITTestVariantCrossEngineCompatibility {
 
     tableEnv.executeSql(createTableDdl);
 
-    // Query the table to verify Flink can read the data
     TableResult result = tableEnv.executeSql("SELECT id, name, v, ts FROM variant_table ORDER BY id");
     List<Row> rows = CollectionUtil.iteratorToList(result.collect());
 
diff --git a/hudi-flink-datasource/hudi-flink1.17.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink1.17.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
index 79b9e254dd61..e8e31b341a18 100644
--- a/hudi-flink-datasource/hudi-flink1.17.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
+++ b/hudi-flink-datasource/hudi-flink1.17.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -22,17 +22,27 @@ import org.apache.flink.table.data.RowData;
 import org.apache.flink.table.types.DataType;
 import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.types.variant.Variant;
+import org.apache.hudi.common.util.Option;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
 
 /**
  * Adapter utils to provide {@code DataType} utilities.
  */
 public class DataTypeAdapter {
+  private static final String VARIANT_UNSUPPORTED_MSG =
+      "VARIANT type is only supported in Flink 2.1+. "
+          + "Please upgrade your Flink version to use Variant columns.";
+
+  public static Option<LogicalTypeAnnotation> variantParquetAnnotation() {
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
+  }
+
   public static Variant getVariant(RowData rowData, int pos) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static Object createVariant(byte[] value, byte[] metadata) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static boolean isVariantType(LogicalType logicalType) {
@@ -40,14 +50,14 @@ public class DataTypeAdapter {
   }
 
   public static DataType createVariantType() {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantMetadata(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantValue(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 }
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
index 79b9e254dd61..e8e31b341a18 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -22,17 +22,27 @@ import org.apache.flink.table.data.RowData;
 import org.apache.flink.table.types.DataType;
 import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.types.variant.Variant;
+import org.apache.hudi.common.util.Option;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
 
 /**
  * Adapter utils to provide {@code DataType} utilities.
  */
 public class DataTypeAdapter {
+  private static final String VARIANT_UNSUPPORTED_MSG =
+      "VARIANT type is only supported in Flink 2.1+. "
+          + "Please upgrade your Flink version to use Variant columns.";
+
+  public static Option<LogicalTypeAnnotation> variantParquetAnnotation() {
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
+  }
+
   public static Variant getVariant(RowData rowData, int pos) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static Object createVariant(byte[] value, byte[] metadata) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static boolean isVariantType(LogicalType logicalType) {
@@ -40,14 +50,14 @@ public class DataTypeAdapter {
   }
 
   public static DataType createVariantType() {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantMetadata(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantValue(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 }
diff --git a/hudi-flink-datasource/hudi-flink1.19.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink1.19.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
index 79b9e254dd61..e8e31b341a18 100644
--- a/hudi-flink-datasource/hudi-flink1.19.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
+++ b/hudi-flink-datasource/hudi-flink1.19.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -22,17 +22,27 @@ import org.apache.flink.table.data.RowData;
 import org.apache.flink.table.types.DataType;
 import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.types.variant.Variant;
+import org.apache.hudi.common.util.Option;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
 
 /**
  * Adapter utils to provide {@code DataType} utilities.
  */
 public class DataTypeAdapter {
+  private static final String VARIANT_UNSUPPORTED_MSG =
+      "VARIANT type is only supported in Flink 2.1+. "
+          + "Please upgrade your Flink version to use Variant columns.";
+
+  public static Option<LogicalTypeAnnotation> variantParquetAnnotation() {
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
+  }
+
   public static Variant getVariant(RowData rowData, int pos) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static Object createVariant(byte[] value, byte[] metadata) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static boolean isVariantType(LogicalType logicalType) {
@@ -40,14 +50,14 @@ public class DataTypeAdapter {
   }
 
   public static DataType createVariantType() {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantMetadata(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantValue(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 }
diff --git a/hudi-flink-datasource/hudi-flink1.20.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink1.20.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
index 79b9e254dd61..e8e31b341a18 100644
--- a/hudi-flink-datasource/hudi-flink1.20.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
+++ b/hudi-flink-datasource/hudi-flink1.20.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -22,17 +22,27 @@ import org.apache.flink.table.data.RowData;
 import org.apache.flink.table.types.DataType;
 import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.types.variant.Variant;
+import org.apache.hudi.common.util.Option;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
 
 /**
  * Adapter utils to provide {@code DataType} utilities.
  */
 public class DataTypeAdapter {
+  private static final String VARIANT_UNSUPPORTED_MSG =
+      "VARIANT type is only supported in Flink 2.1+. "
+          + "Please upgrade your Flink version to use Variant columns.";
+
+  public static Option<LogicalTypeAnnotation> variantParquetAnnotation() {
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
+  }
+
   public static Variant getVariant(RowData rowData, int pos) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static Object createVariant(byte[] value, byte[] metadata) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static boolean isVariantType(LogicalType logicalType) {
@@ -40,14 +50,14 @@ public class DataTypeAdapter {
   }
 
   public static DataType createVariantType() {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantMetadata(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantValue(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 }
diff --git a/hudi-flink-datasource/hudi-flink2.0.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink2.0.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
index 566ca723648a..18db48f0fed1 100644
--- a/hudi-flink-datasource/hudi-flink2.0.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
+++ b/hudi-flink-datasource/hudi-flink2.0.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -22,17 +22,27 @@ import org.apache.flink.table.data.RowData;
 import org.apache.flink.table.types.DataType;
 import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.types.variant.Variant;
+import org.apache.hudi.common.util.Option;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
 
 /**
  * Adapter utils to provide {@code DataType} utilities.
  */
 public class DataTypeAdapter {
+  private static final String VARIANT_UNSUPPORTED_MSG =
+      "VARIANT type is only supported in Flink 2.1+. "
+          + "Please upgrade your Flink version to use Variant columns.";
+
+  public static Option<LogicalTypeAnnotation> variantParquetAnnotation() {
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
+  }
+
   public static Variant getVariant(RowData rowData, int pos) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static Object createVariant(byte[] value, byte[] metadata) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static boolean isVariantType(LogicalType logicalType) {
@@ -40,14 +50,14 @@ public class DataTypeAdapter {
   }
 
   public static DataType createVariantType() {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantMetadata(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 
   public static byte[] getVariantValue(Object obj) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    throw new UnsupportedOperationException(VARIANT_UNSUPPORTED_MSG);
   }
 }
\ No newline at end of file
diff --git a/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
index 6ae4c4a6baba..77d6f708ece3 100644
--- a/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
+++ b/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -25,11 +25,46 @@ import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.table.types.logical.LogicalTypeRoot;
 import org.apache.flink.types.variant.BinaryVariant;
 import org.apache.flink.types.variant.Variant;
+import org.apache.hudi.common.util.Option;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+
+import java.lang.reflect.Method;
 
 /**
  * Adapter utils to provide {@code DataType} utilities.
  */
 public class DataTypeAdapter {
+
+  /**
+   * The Parquet Variant binary format specification version passed to
+   * {@code LogicalTypeAnnotation.variantType(byte)}. Version 1 is the initial spec
+   * defined by the Parquet Variant proposal (parquet-format 2.11.0 / parquet-java 1.16.0).
+   */
+  private static final byte VARIANT_SPEC_VERSION = 1;
+
+  /**
+   * Cached VARIANT annotation resolved via reflection. Empty if parquet-java
+   * on the classpath predates {@code LogicalTypeAnnotation.variantType()} (< 1.16.0).
+   */
+  private static final Option<LogicalTypeAnnotation> VARIANT_ANNOTATION = resolveVariantAnnotation();
+
+  private static Option<LogicalTypeAnnotation> resolveVariantAnnotation() {
+    try {
+      Method factory = LogicalTypeAnnotation.class.getMethod("variantType", byte.class);
+      return Option.of((LogicalTypeAnnotation) factory.invoke(null, VARIANT_SPEC_VERSION));
+    } catch (Exception e) {
+      return Option.empty();
+    }
+  }
+
+  /**
+   * Returns the Parquet VARIANT {@link LogicalTypeAnnotation} if parquet-java 1.16.0+ is on the
+   * classpath, or empty if the annotation class is unavailable.
+   */
+  public static Option<LogicalTypeAnnotation> variantParquetAnnotation() {
+    return VARIANT_ANNOTATION;
+  }
+
   public static Variant getVariant(RowData rowData, int pos) {
     return rowData.getVariant(pos);
   }


(hudi) 08/16: docs: Document muttley package as internal/optional for OSS users (#18394)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 276565468ef017609ef0dad82348225b9bc5eaa6
Author: Xinli Shang <sh...@uber.com>
AuthorDate: Sat May 9 19:10:17 2026 -0700

    docs: Document muttley package as internal/optional for OSS users (#18394)
    
    (cherry picked from commit f8e4b9f3db52fcaab05a37b15f7ea7ce8c8b3355)
---
 .../java/org/apache/hudi/sink/muttley/README.md    | 25 ++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/muttley/README.md b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/muttley/README.md
new file mode 100644
index 000000000000..04c2e31d0e1e
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/muttley/README.md
@@ -0,0 +1,25 @@
+# Internal Uber Components (Optional)
+
+This package contains integration with internal Uber services and is **optional** for Apache Hudi users.
+
+## Components
+
+- **FlinkHudiMuttleyClient** - Abstract base class providing HTTP retry logic for Muttley RPC communication
+- **AthenaIngestionGateway** - Concrete Muttley RPC client for Uber's Athena Ingestion Gateway service (extends `FlinkHudiMuttleyClient`)
+- **FlinkHudiMuttleyException** - Base exception for Muttley errors
+- **FlinkHudiMuttleyClientException** - Exception for client-side Muttley errors
+- **FlinkHudiMuttleyServerException** - Exception for server-side Muttley errors
+
+## Usage
+
+These components are used by `FlinkCheckpointClient` to collect Kafka offset metadata and attach it to Hudi commits as part of Uber's Kafka offset tracking feature.
+
+**Feature flag**: The feature is controlled by `write.extra.metadata.enabled` (default: `false`). It is disabled by default and has no effect in standard Apache Hudi deployments.
+
+**For open-source Apache Hudi users**: You can safely ignore this package. If `write.extra.metadata.enabled` is inadvertently set to `true` outside of Uber's infrastructure, Kafka offset collection will be silently skipped and Hudi commits will proceed normally (fail-open behavior).
+
+## Dependencies
+
+- Runtime: Uber's internal Muttley RPC framework
+- Runtime: Access to Athena Ingestion Gateway service
+- Compile-time: Jackson (`jackson-databind`) for JSON serialization of RPC request/response payloads


(hudi) 13/16: feat(utilities): add Spark/HoodieStreamer validators for pre-commit validation - Phase 3 (#18405)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 64974eb0e9e9a947ca0e1184d9a1f98865998a14
Author: Xinli Shang <sh...@uber.com>
AuthorDate: Sat May 16 20:37:56 2026 -0700

    feat(utilities): add Spark/HoodieStreamer validators for pre-commit validation - Phase 3 (#18405)
    
    * feat: Add Spark streamer validators for phase 3 precommit validation
    
    Implements phase 3 of the precommit validation framework by adding:
    - SparkKafkaOffsetValidator: Validates Kafka offset consistency
    - SparkValidationContext: Provides Spark-specific validation context
    - SparkStreamerValidatorUtils: Utility functions for Spark streamer validation
    - Comprehensive test coverage for all validator components
    - Integration with StreamSync and HoodiePreCommitValidatorConfig
    
    Co-Authored-By: Claude Opus 4.6 <no...@anthropic.com>
    
    * fix: address code review and fix checkstyle violations
    
    - Remove unused imports (java.io.IOException, HoodieCommitMetadata,
      HoodieTestTable, Option) that caused checkstyle build failures
    - Remove accidentally committed bootstrap_register_only_issue.md
    - Cache writeStatusRDD before collect() to prevent second DAG evaluation
      and potential driver OOM
    - Add comment explaining why validator runs before writeClient.commit():
      offset validation is a stronger guard than commitOnErrors and must
      prevent the commit when data loss is detected
    - Clarify buildCommitMetadata() produces a pre-commit preview object,
      not a fully-constructed commit record
    - Add Javadoc to SparkKafkaOffsetValidator and SparkStreamerValidatorUtils
      explaining incompatibility with SparkValidatorUtils (different interface
      and constructor signature) to prevent misconfiguration
    - Add two-commit integration tests (testSecondCommitMatchingOffsetsPasses,
      testSecondCommitDataLossDetected) using HoodieTestTable to exercise the
      real offset comparison path, not just the first-commit skip path
    
    * fix: skip non-SparkPreCommitValidator classes in SparkValidatorUtils
    
    SparkKafkaOffsetValidator (and similar streaming validators) extend
    BasePreCommitValidator with a (TypedProperties) constructor, not the
    (HoodieSparkTable, HoodieEngineContext, HoodieWriteConfig) constructor
    that SparkValidatorUtils expects. Listing such a validator in
    hoodie.precommit.validators previously caused a reflection error in the
    Spark table write path.
    
    Add a Class.isAssignableFrom check to filter out classes that don't
    implement SparkPreCommitValidator before attempting instantiation, with
    a clear warning pointing users to SparkStreamerValidatorUtils for
    streaming validators.
    
    * ci: trigger CI re-run for flaky trino test
    
    * fix: address reviewer comments on pre-commit streaming offset validator
    
    - Unpersist cached RDD in finally block to prevent executor memory leak
    - Let IOException propagate from loadPreviousCommitMetadata instead of silently swallowing it
    - Filter empty validator class names before Class.forName to handle trailing comma in config
    - Add write error count to validation message to distinguish write failures from silent data loss
    
    * fix: address reviewer follow-up comments on pre-commit streaming validator
    
    - Change runValidators to accept List<WriteStatus> instead of JavaRDD
      to fix RDD unpersist-before-commit bug; StreamSync now caches the RDD,
      collects to list for validators, passes RDD to commit, then unpersists
    
    - Remove generic catch(Exception) in loadPreviousCommitMetadata so
      non-IOException failures propagate instead of silently skipping validation
    
    - Implement getPreviousCommitInstant() in SparkValidationContext via
      timeline lookup instead of throwing UnsupportedOperationException
    
    - Add Objects::nonNull filter when building writeStats list
    
    - Add BasePreCommitValidator assignability guard in SparkStreamerValidatorUtils
      to warn and skip SparkPreCommitValidator classes (reverse-direction guard)
    
    - Eliminate double class loading in SparkValidatorUtils by combining
      filter+map into a single flatMap; remove unused ReflectionUtils import
    
    - Remove trivial constructor Javadoc from SparkKafkaOffsetValidator
    
    - Add HoodieTestUtils import in test; remove Spark context boilerplate
      now that runValidators accepts List<WriteStatus> directly
    
    * fix: guard cache() call when writeStatusRDD is already persisted
    
    Calling cache() on an RDD that already has a storage level assigned throws
    SparkUnsupportedOperationException. The write path may cache the RDD
    internally before returning it. Track whether we own the cache and only
    call cache()/unpersist() when the RDD was not already persisted.
    
    * Ensure writeStatusRDD is always unpersisted via try/finally
    
    * fix: use proper import for StorageLevel instead of fully-qualified class reference
    
    πŸ€– Generated with [Claude Code](https://claude.ai/code)
    
    Co-Authored-By: Claude Sonnet 4.6 <no...@anthropic.com>
    
    * fix: address final reviewer feedback on pre-commit streaming validator
    
    - Move runValidators() inside the try/finally so writeStatusRDD.unpersist()
      always runs, including on validator exceptions (FAIL policy or
      HoodieIOException from loadPreviousCommitMetadata).
    - Use ReflectionUtils.loadClass in SparkValidatorUtils for instantiation,
      matching SparkStreamerValidatorUtils and the rest of the codebase.
    - Rename weOwnCache to shouldUnpersist to read in the direction of its
      actual use (gating unpersist in finally).
    
    * fix: only cache writeStatusRDD when pre-commit validators are configured
    
    Address danny0405's review comment on PR #18405: skip the .cache()/.unpersist()
    cycle when no pre-commit validators are configured, since without validators the
    RDD is consumed exactly once by writeClient.commit() and caching adds no value.
    
    Guards both the cache call and the validator collect+run on a single
    validatorsConfigured boolean derived from hoodie.precommit.validators.
    
    * address codope review: V2-then-V1 checkpoint key resolution + V2 test coverage
    
    Comment 1 (SparkKafkaOffsetValidator hardcoded V1 key):
    - StreamingOffsetValidator base class now exposes a no-key constructor that
      auto-resolves the checkpoint via CheckpointUtils.getCheckpoint(metadata),
      which prefers V2 and falls back to V1. The explicit-key constructor stays
      for subclasses that read a custom non-streamer key (e.g. Flink's
      HOODIE_METADATA_KEY).
    - SparkKafkaOffsetValidator switches to the no-key constructor.
    
    Comment 2 (tests only cover V1 path):
    - testSecondCommitMatchingOffsetsPasses and testSecondCommitDataLossDetected
      are now parameterized over both V1 and V2 checkpoint keys.
    - Added testV2CheckpointKeyOnTableVersionEightFires on a HoodieTableVersion.EIGHT
      table with V2 keys, asserting the validator fires on data loss.
    
    ---------
    
    Co-authored-by: Xinli Shang <sh...@apache.org>
    Co-authored-by: Claude Opus 4.6 <no...@anthropic.com>
    (cherry picked from commit b934633f066a6efa57e04837b2536baa66b4f2f4)
---
 .../client/validator/StreamingOffsetValidator.java |  93 +++++-
 .../config/HoodiePreCommitValidatorConfig.java     |   8 +-
 .../validator/TestStreamingOffsetValidator.java    |   2 +-
 .../hudi/client/utils/SparkValidatorUtils.java     |  25 +-
 .../hudi/client/validator/ValidationContext.java   |  14 +
 .../apache/hudi/utilities/streamer/StreamSync.java |  37 ++-
 .../validator/SparkKafkaOffsetValidator.java       |  60 ++++
 .../validator/SparkStreamerValidatorUtils.java     | 194 +++++++++++++
 .../streamer/validator/SparkValidationContext.java | 139 +++++++++
 .../validator/TestSparkKafkaOffsetValidator.java   | 322 +++++++++++++++++++++
 .../validator/TestSparkStreamerValidatorUtils.java | 290 +++++++++++++++++++
 .../validator/TestSparkValidationContext.java      | 156 ++++++++++
 12 files changed, 1317 insertions(+), 23 deletions(-)

diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/validator/StreamingOffsetValidator.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/validator/StreamingOffsetValidator.java
index ce577d84ca01..40e7a4f1635f 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/validator/StreamingOffsetValidator.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/validator/StreamingOffsetValidator.java
@@ -20,11 +20,13 @@
 package org.apache.hudi.client.validator;
 
 import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
 import org.apache.hudi.common.util.CheckpointUtils;
 import org.apache.hudi.common.util.CheckpointUtils.CheckpointFormat;
 import org.apache.hudi.common.util.Option;
 import org.apache.hudi.config.HoodiePreCommitValidatorConfig;
 import org.apache.hudi.config.HoodiePreCommitValidatorConfig.ValidationFailurePolicy;
+import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.exception.HoodieValidationException;
 
 import lombok.extern.slf4j.Slf4j;
@@ -50,7 +52,11 @@ import lombok.extern.slf4j.Slf4j;
  *
  * Subclasses specify:
  * - Checkpoint format (SPARK_KAFKA, FLINK_KAFKA, etc.)
- * - Checkpoint metadata key
+ * - Checkpoint metadata key (optional β€” when omitted, the validator auto-resolves the
+ *   active streamer key from commit metadata using
+ *   {@link org.apache.hudi.common.table.checkpoint.CheckpointUtils#getCheckpoint(HoodieCommitMetadata)},
+ *   which prefers V2 and falls back to V1. Subclasses that read a custom non-streamer key
+ *   (e.g. Flink's HOODIE_METADATA_KEY) must pass it explicitly.)
  * - Source-specific parsing logic (if needed)
  *
  * Configuration:
@@ -66,7 +72,26 @@ public abstract class StreamingOffsetValidator extends BasePreCommitValidator {
   protected final CheckpointFormat checkpointFormat;
 
   /**
-   * Create a streaming offset validator.
+   * Create a streaming offset validator that auto-resolves the checkpoint key from commit
+   * metadata using {@link org.apache.hudi.common.table.checkpoint.CheckpointUtils#getCheckpoint(HoodieCommitMetadata)}.
+   *
+   * <p>Use this constructor for streamer pipelines (V1 or V2 checkpoint keys). The validator
+   * will prefer V2 (table version 8+) and fall back to V1 transparently, so subclasses don't
+   * need to know which key the writer used.</p>
+   *
+   * @param config Validator configuration
+   * @param checkpointFormat Format of the checkpoint string
+   */
+  protected StreamingOffsetValidator(TypedProperties config,
+                                      CheckpointFormat checkpointFormat) {
+    this(config, null, checkpointFormat);
+  }
+
+  /**
+   * Create a streaming offset validator with an explicit checkpoint metadata key.
+   *
+   * <p>Use this constructor when the writer stores its checkpoint under a custom key that
+   * is not the standard streamer V1/V2 key (e.g. Flink's HOODIE_METADATA_KEY).</p>
    *
    * @param config Validator configuration
    * @param checkpointKey Key to extract checkpoint from extraMetadata
@@ -95,10 +120,12 @@ public abstract class StreamingOffsetValidator extends BasePreCommitValidator {
       return;
     }
 
-    // Extract current checkpoint
-    Option<String> currentCheckpointOpt = context.getExtraMetadata(checkpointKey);
+    // Extract current checkpoint β€” either from the explicit key (custom writers like Flink) or
+    // by auto-resolving from commit metadata (streamer pipelines, V2-then-V1 fallback).
+    Option<String> currentCheckpointOpt = resolveCheckpoint(context.getCommitMetadata());
     if (!currentCheckpointOpt.isPresent()) {
-      log.warn("Current checkpoint not found with key: {}. Skipping validation.", checkpointKey);
+      log.warn("Current checkpoint not found (key: {}). Skipping validation.",
+          checkpointKey == null ? "<auto-resolved>" : checkpointKey);
       return;
     }
     String currentCheckpoint = currentCheckpointOpt.get();
@@ -110,8 +137,7 @@ public abstract class StreamingOffsetValidator extends BasePreCommitValidator {
     }
 
     // Extract previous checkpoint
-    Option<String> previousCheckpointOpt = context.getPreviousCommitMetadata()
-        .flatMap(metadata -> Option.ofNullable(metadata.getMetadata(checkpointKey)));
+    Option<String> previousCheckpointOpt = resolveCheckpoint(context.getPreviousCommitMetadata());
 
     if (!previousCheckpointOpt.isPresent()) {
       log.info("Previous checkpoint not found. May be first streaming commit. Skipping validation.");
@@ -139,6 +165,10 @@ public abstract class StreamingOffsetValidator extends BasePreCommitValidator {
     long recordsWritten = context.getTotalInsertRecordsWritten()
         + context.getTotalUpdateRecordsWritten();
 
+    // Track write errors so callers can distinguish write-failure deviation (write errors > 0)
+    // from silent data loss (write errors == 0) when the validator fires.
+    long writeErrors = context.getTotalWriteErrors();
+
     // For empty commits (e.g., no new data from source), both offsetDiff and recordsWritten
     // can be zero. This is a valid scenario β€” skip validation to avoid false positives.
     if (offsetDifference == 0 && recordsWritten == 0) {
@@ -147,7 +177,7 @@ public abstract class StreamingOffsetValidator extends BasePreCommitValidator {
     }
 
     // Validate offset vs record consistency
-    validateOffsetConsistency(offsetDifference, recordsWritten,
+    validateOffsetConsistency(offsetDifference, recordsWritten, writeErrors,
         currentCheckpoint, previousCheckpoint);
   }
 
@@ -155,12 +185,13 @@ public abstract class StreamingOffsetValidator extends BasePreCommitValidator {
    * Validate that offset difference matches record count within tolerance.
    *
    * @param offsetDiff Expected records based on offset difference
-   * @param recordsWritten Actual records written
+   * @param recordsWritten Actual records written (inserts + updates)
+   * @param writeErrors Records that failed to write (tracked in write status errors)
    * @param currentCheckpoint Current checkpoint string (for error messages)
    * @param previousCheckpoint Previous checkpoint string (for error messages)
    * @throws HoodieValidationException if validation fails and policy is FAIL
    */
-  protected void validateOffsetConsistency(long offsetDiff, long recordsWritten,
+  protected void validateOffsetConsistency(long offsetDiff, long recordsWritten, long writeErrors,
                                             String currentCheckpoint, String previousCheckpoint)
       throws HoodieValidationException {
 
@@ -169,10 +200,13 @@ public abstract class StreamingOffsetValidator extends BasePreCommitValidator {
     if (deviation > tolerancePercentage) {
       String errorMsg = String.format(
           "Streaming offset validation failed. "
-              + "Offset difference: %d, Records written: %d, Deviation: %.2f%%, Tolerance: %.2f%%. "
-              + "This may indicate data loss or filtering. "
+              + "Offset difference: %d, Records written: %d, Write errors: %d, Deviation: %.2f%%, Tolerance: %.2f%%. "
+              + "%s"
               + "Previous checkpoint: %s, Current checkpoint: %s",
-          offsetDiff, recordsWritten, deviation, tolerancePercentage,
+          offsetDiff, recordsWritten, writeErrors, deviation, tolerancePercentage,
+          writeErrors > 0
+              ? "Non-zero write errors suggest records failed to write rather than silent data loss. "
+              : "This may indicate data loss or filtering. ",
           previousCheckpoint, currentCheckpoint);
 
       if (failurePolicy == ValidationFailurePolicy.WARN_LOG) {
@@ -181,8 +215,8 @@ public abstract class StreamingOffsetValidator extends BasePreCommitValidator {
         throw new HoodieValidationException(errorMsg);
       }
     } else {
-      log.info("Offset validation passed. Offset diff: {}, Records: {}, Deviation: {}% (within {}%)",
-          offsetDiff, recordsWritten, String.format("%.2f", deviation), tolerancePercentage);
+      log.info("Offset validation passed. Offset diff: {}, Records: {}, Write errors: {}, Deviation: {}% (within {}%)",
+          offsetDiff, recordsWritten, writeErrors, String.format("%.2f", deviation), tolerancePercentage);
     }
   }
 
@@ -210,4 +244,33 @@ public abstract class StreamingOffsetValidator extends BasePreCommitValidator {
     long difference = Math.abs(offsetDiff - recordsWritten);
     return (100.0 * difference) / offsetDiff;
   }
+
+  /**
+   * Resolve the checkpoint string from commit metadata.
+   *
+   * <p>When the validator was constructed with an explicit {@code checkpointKey}, that key
+   * is read directly. Otherwise, {@link org.apache.hudi.common.table.checkpoint.CheckpointUtils#getCheckpoint(HoodieCommitMetadata)}
+   * is used to locate the active streamer checkpoint (V2 first, V1 fallback), so callers
+   * don't need to know which key the writer used.</p>
+   *
+   * @param commitMetadataOpt Optional commit metadata containing extraMetadata
+   * @return Optional checkpoint string (empty if metadata is absent or no checkpoint key matches)
+   */
+  private Option<String> resolveCheckpoint(Option<HoodieCommitMetadata> commitMetadataOpt) {
+    if (!commitMetadataOpt.isPresent()) {
+      return Option.empty();
+    }
+    HoodieCommitMetadata metadata = commitMetadataOpt.get();
+    if (checkpointKey != null) {
+      return Option.ofNullable(metadata.getMetadata(checkpointKey));
+    }
+    try {
+      return Option.ofNullable(
+          org.apache.hudi.common.table.checkpoint.CheckpointUtils.getCheckpoint(metadata)
+              .getCheckpointKey());
+    } catch (HoodieException e) {
+      // No V1 or V2 streamer checkpoint key present in extraMetadata.
+      return Option.empty();
+    }
+  }
 }
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodiePreCommitValidatorConfig.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodiePreCommitValidatorConfig.java
index f85cc44120d4..169494b7244a 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodiePreCommitValidatorConfig.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodiePreCommitValidatorConfig.java
@@ -43,7 +43,10 @@ public class HoodiePreCommitValidatorConfig extends HoodieConfig {
       .key("hoodie.precommit.validators")
       .defaultValue("")
       .markAdvanced()
-      .withDocumentation("Comma separated list of class names that can be invoked to validate commit");
+      .withDocumentation("Comma separated list of class names that can be invoked to validate commit. "
+          + "Available streaming offset validators: "
+          + "org.apache.hudi.sink.validator.FlinkKafkaOffsetValidator (Flink Kafka), "
+          + "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator (Spark/HoodieStreamer Kafka)");
   public static final String VALIDATOR_TABLE_VARIABLE = "<TABLE_NAME>";
 
   public static final ConfigProperty<String> EQUALITY_SQL_QUERIES = ConfigProperty
@@ -71,7 +74,8 @@ public class HoodiePreCommitValidatorConfig extends HoodieConfig {
       .markAdvanced()
       .withDocumentation("Tolerance percentage for streaming offset validation "
           + "(used by org.apache.hudi.client.validator.StreamingOffsetValidator "
-          + "and org.apache.hudi.sink.validator.FlinkKafkaOffsetValidator). "
+          + "and org.apache.hudi.sink.validator.FlinkKafkaOffsetValidator "
+          + "and org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator). "
           + "The validator compares the offset difference (expected records from source) "
           + "with actual records written. If the deviation exceeds this percentage, "
           + "the commit is rejected or warned depending on the validation failure policy. "
diff --git a/hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/validator/TestStreamingOffsetValidator.java b/hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/validator/TestStreamingOffsetValidator.java
index 818bc2087a3c..bc402e5ee5a0 100644
--- a/hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/validator/TestStreamingOffsetValidator.java
+++ b/hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/validator/TestStreamingOffsetValidator.java
@@ -62,7 +62,7 @@ public class TestStreamingOffsetValidator {
     // Expose protected method for testing
     public void testValidateOffsetConsistency(long offsetDiff, long recordsWritten,
                                                 String current, String previous) {
-      validateOffsetConsistency(offsetDiff, recordsWritten, current, previous);
+      validateOffsetConsistency(offsetDiff, recordsWritten, 0L, current, previous);
     }
 
     // Expose validateWithMetadata for testing
diff --git a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkValidatorUtils.java b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkValidatorUtils.java
index c7804631d3f6..40ee7d8cefff 100644
--- a/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkValidatorUtils.java
+++ b/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkValidatorUtils.java
@@ -84,9 +84,28 @@ public class SparkValidatorUtils {
       Dataset<Row> beforeState = getRecordsFromCommittedFiles(sqlContext, partitionsModified, table, afterState.schema());
 
       Stream<SparkPreCommitValidator> validators = Arrays.stream(config.getPreCommitValidators().split(","))
-          .map(validatorClass -> ((SparkPreCommitValidator) ReflectionUtils.loadClass(validatorClass,
-              new Class<?>[] {HoodieSparkTable.class, HoodieEngineContext.class, HoodieWriteConfig.class},
-              table, context, config)));
+          .map(String::trim)
+          .filter(s -> !s.isEmpty())
+          .flatMap(validatorClass -> {
+            try {
+              Class<?> clazz = Class.forName(validatorClass);
+              if (!SparkPreCommitValidator.class.isAssignableFrom(clazz)) {
+                LOG.warn("Skipping validator {} β€” it does not implement SparkPreCommitValidator. "
+                    + "If this is a streaming offset validator (e.g. SparkKafkaOffsetValidator), "
+                    + "it will be invoked by SparkStreamerValidatorUtils instead.", validatorClass);
+                return Stream.empty();
+              }
+              SparkPreCommitValidator validator = (SparkPreCommitValidator) ReflectionUtils.loadClass(
+                  validatorClass,
+                  new Class<?>[] {HoodieSparkTable.class, HoodieEngineContext.class, HoodieWriteConfig.class},
+                  table, context, config);
+              return Stream.of(validator);
+            } catch (ClassNotFoundException e) {
+              throw new HoodieValidationException("Cannot find validator class: " + validatorClass, e);
+            } catch (ReflectiveOperationException e) {
+              throw new HoodieValidationException("Failed to instantiate validator: " + validatorClass, e);
+            }
+          });
 
       boolean allSuccess = validators.map(v -> runValidatorAsync(v, writeMetadata, beforeState, afterState, instantTime)).map(CompletableFuture::join)
           .reduce(true, Boolean::logicalAnd);
diff --git a/hudi-common/src/main/java/org/apache/hudi/client/validator/ValidationContext.java b/hudi-common/src/main/java/org/apache/hudi/client/validator/ValidationContext.java
index 30fdbb3ba3b4..b85218e587e8 100644
--- a/hudi-common/src/main/java/org/apache/hudi/client/validator/ValidationContext.java
+++ b/hudi-common/src/main/java/org/apache/hudi/client/validator/ValidationContext.java
@@ -169,6 +169,20 @@ public interface ValidationContext {
         .orElse(0L);
   }
 
+  /**
+   * Calculate total write errors in the current commit.
+   * Records that failed to write are tracked in {@link org.apache.hudi.common.model.HoodieWriteStat#getTotalWriteErrors()}.
+   * A non-zero error count alongside a deviation in offset validation indicates write failures
+   * rather than silent data loss β€” useful context for distinguishing the two failure modes.
+   *
+   * @return Total count of records that failed to write
+   */
+  default long getTotalWriteErrors() {
+    return getWriteStats()
+        .map(stats -> stats.stream().mapToLong(HoodieWriteStat::getTotalWriteErrors).sum())
+        .orElse(0L);
+  }
+
   /**
    * Check if this is the first commit (no previous commits exist).
    * Derived from {@link #getPreviousCommitInstant()}.
diff --git a/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java
index 600891c85dff..2b1406c7deab 100644
--- a/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java
+++ b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java
@@ -74,6 +74,7 @@ import org.apache.hudi.config.HoodieCompactionConfig;
 import org.apache.hudi.config.HoodieErrorTableConfig;
 import org.apache.hudi.config.HoodieIndexConfig;
 import org.apache.hudi.config.HoodiePayloadConfig;
+import org.apache.hudi.config.HoodiePreCommitValidatorConfig;
 import org.apache.hudi.config.HoodieWriteConfig;
 import org.apache.hudi.config.metrics.HoodieMetricsConfig;
 import org.apache.hudi.data.HoodieJavaRDD;
@@ -115,6 +116,7 @@ import org.apache.hudi.utilities.schema.SimpleSchemaProvider;
 import org.apache.hudi.utilities.sources.InputBatch;
 import org.apache.hudi.utilities.sources.Source;
 import org.apache.hudi.utilities.streamer.HoodieStreamer.Config;
+import org.apache.hudi.utilities.streamer.validator.SparkStreamerValidatorUtils;
 import org.apache.hudi.utilities.transform.Transformer;
 
 import com.codahale.metrics.Timer;
@@ -128,6 +130,7 @@ import org.apache.spark.sql.Dataset;
 import org.apache.spark.sql.Row;
 import org.apache.spark.sql.SparkSession;
 import org.apache.spark.sql.types.StructType;
+import org.apache.spark.storage.StorageLevel;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -874,8 +877,38 @@ public class StreamSync implements Serializable, Closeable {
           totalSuccessfulRecords);
       String commitActionType = CommitUtils.getCommitActionType(cfg.operation, HoodieTableType.valueOf(cfg.tableType));
 
-      boolean success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata), commitActionType, partitionToReplacedFileIds, Option.empty(),
-          Option.of(writeStatusValidator));
+      // Cache the RDD only when pre-commit validators are configured. Validators collect the RDD
+      // before commit, so without caching the same DAG would re-evaluate inside writeClient.commit().
+      // When no validators are configured, commit consumes the RDD once and caching adds no value.
+      // shouldUnpersist is true only when we created the cache here (validators present and storage
+      // level was NONE), so the finally block knows to release it.
+      boolean validatorsConfigured = !StringUtils.isNullOrEmpty(props.getString(
+          HoodiePreCommitValidatorConfig.VALIDATOR_CLASS_NAMES.key(),
+          HoodiePreCommitValidatorConfig.VALIDATOR_CLASS_NAMES.defaultValue()));
+      boolean shouldUnpersist = validatorsConfigured && writeStatusRDD.getStorageLevel().equals(StorageLevel.NONE());
+      if (shouldUnpersist) {
+        writeStatusRDD.cache();
+      }
+      boolean success;
+      try {
+        if (validatorsConfigured) {
+          List<WriteStatus> writeStatuses = writeStatusRDD.collect();
+
+          // Run pre-commit streaming offset validators (if configured).
+          // Placement before writeClient.commit() is intentional: offset validation is a stronger
+          // guard than commitOnErrors β€” if offset deviation indicates potential data loss, the commit
+          // must be prevented regardless of the commitOnErrors policy.
+          SparkStreamerValidatorUtils.runValidators(props, instantTime, writeStatuses,
+              checkpointCommitMetadata, metaClient);
+        }
+
+        success = writeClient.commit(instantTime, writeStatusRDD, Option.of(checkpointCommitMetadata), commitActionType, partitionToReplacedFileIds, Option.empty(),
+            Option.of(writeStatusValidator));
+      } finally {
+        if (shouldUnpersist) {
+          writeStatusRDD.unpersist();
+        }
+      }
       releaseResourcesInvoked = true;
       if (success) {
         LOG.info("Commit " + instantTime + " successful!");
diff --git a/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkKafkaOffsetValidator.java b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkKafkaOffsetValidator.java
new file mode 100644
index 000000000000..589e09e96de8
--- /dev/null
+++ b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkKafkaOffsetValidator.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.streamer.validator;
+
+import org.apache.hudi.client.validator.StreamingOffsetValidator;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.CheckpointUtils.CheckpointFormat;
+
+/**
+ * Spark/HoodieStreamer-specific Kafka offset validator.
+ *
+ * <p>Validates that the number of records written matches the Kafka offset difference
+ * between the current and previous HoodieStreamer checkpoints. The active checkpoint key
+ * (V1 {@code deltastreamer.checkpoint.key} or V2 {@code streamer.checkpoint.key.v2}) is
+ * resolved at validation time via {@link org.apache.hudi.common.table.checkpoint.CheckpointUtils#getCheckpoint},
+ * so this validator works against tables written with either checkpoint key version.</p>
+ *
+ * <p>Configuration:
+ * <ul>
+ *   <li>{@code hoodie.precommit.validators}: Include
+ *       {@code org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator}</li>
+ *   <li>{@code hoodie.precommit.validators.streaming.offset.tolerance.percentage}:
+ *       Acceptable deviation (default: 0.0 = strict)</li>
+ *   <li>{@code hoodie.precommit.validators.failure.policy}:
+ *       FAIL (default) or WARN_LOG</li>
+ * </ul></p>
+ *
+ * <p>This validator is primarily intended for append-only ingestion from Kafka via HoodieStreamer.
+ * For upsert workloads with deduplication, configure a higher tolerance or use WARN_LOG.</p>
+ *
+ * <p><b>Important:</b> This class extends {@link org.apache.hudi.client.validator.BasePreCommitValidator}
+ * and is invoked by {@link SparkStreamerValidatorUtils}, NOT by {@code SparkValidatorUtils}
+ * (which expects {@code SparkPreCommitValidator} with a different constructor signature).
+ * Listing this class in {@code hoodie.precommit.validators} while also using the standard
+ * Spark table write-path validators will cause an instantiation failure in {@code SparkValidatorUtils}.
+ * Use this validator exclusively with HoodieStreamer pipelines.</p>
+ */
+public class SparkKafkaOffsetValidator extends StreamingOffsetValidator {
+
+  public SparkKafkaOffsetValidator(TypedProperties config) {
+    super(config, CheckpointFormat.SPARK_KAFKA);
+  }
+}
diff --git a/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkStreamerValidatorUtils.java b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkStreamerValidatorUtils.java
new file mode 100644
index 000000000000..474215ec0f3c
--- /dev/null
+++ b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkStreamerValidatorUtils.java
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.streamer.validator;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.validator.BasePreCommitValidator;
+import org.apache.hudi.client.validator.ValidationContext;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.StringUtils;
+import org.apache.hudi.config.HoodiePreCommitValidatorConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieValidationException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+/**
+ * Utility for running pre-commit validators in the HoodieStreamer commit flow.
+ *
+ * <p>Instantiates and executes validators configured via
+ * {@code hoodie.precommit.validators}. Each validator must extend
+ * {@link BasePreCommitValidator} and have a constructor that accepts
+ * {@link TypedProperties}.</p>
+ *
+ * <p>Called from {@code StreamSync.writeToSinkAndDoMetaSync()} before
+ * the commit is finalized.</p>
+ *
+ * <p><b>Note on validator compatibility:</b> This utility uses a different instantiation
+ * mechanism than {@code SparkValidatorUtils} (used by the Spark table write path).
+ * {@code SparkValidatorUtils} expects validators implementing {@code SparkPreCommitValidator}
+ * with a {@code (HoodieSparkTable, HoodieEngineContext, HoodieWriteConfig)} constructor.
+ * Validators registered here (e.g. {@link SparkKafkaOffsetValidator}) extend
+ * {@link BasePreCommitValidator} with a {@code (TypedProperties)} constructor and
+ * are NOT compatible with {@code SparkValidatorUtils}. Do not mix them under the same
+ * {@code hoodie.precommit.validators} config if both paths are active.</p>
+ */
+public class SparkStreamerValidatorUtils {
+
+  private static final Logger LOG = LoggerFactory.getLogger(SparkStreamerValidatorUtils.class);
+
+  /**
+   * Run all configured pre-commit validators.
+   *
+   * <p>The caller is responsible for caching and unpersisting the source RDD if needed.
+   * This method accepts pre-collected write statuses to avoid a second DAG evaluation β€”
+   * the caller should cache the RDD, collect to this list, call this method, then pass
+   * the same RDD to {@code writeClient.commit()}, and unpersist after commit completes.</p>
+   *
+   * @param props Configuration properties containing validator class names
+   * @param instantTime Commit instant time
+   * @param writeStatuses Pre-collected write statuses from Spark write operations
+   * @param checkpointCommitMetadata Extra metadata being committed (contains checkpoint info)
+   * @param metaClient Table meta client for timeline access and previous commit lookup
+   * @throws HoodieValidationException if any validator fails with FAIL policy
+   */
+  public static void runValidators(TypedProperties props,
+                                   String instantTime,
+                                   List<WriteStatus> writeStatuses,
+                                   Map<String, String> checkpointCommitMetadata,
+                                   HoodieTableMetaClient metaClient) {
+    String validatorClassNames = props.getString(
+        HoodiePreCommitValidatorConfig.VALIDATOR_CLASS_NAMES.key(),
+        HoodiePreCommitValidatorConfig.VALIDATOR_CLASS_NAMES.defaultValue());
+
+    if (StringUtils.isNullOrEmpty(validatorClassNames)) {
+      return;
+    }
+
+    HoodieCommitMetadata currentMetadata = buildCommitMetadata(writeStatuses, checkpointCommitMetadata);
+    List<HoodieWriteStat> writeStats = writeStatuses.stream()
+        .map(WriteStatus::getStat)
+        .filter(Objects::nonNull)
+        .collect(Collectors.toList());
+
+    Option<HoodieCommitMetadata> previousCommitMetadata = loadPreviousCommitMetadata(metaClient);
+
+    ValidationContext context = new SparkValidationContext(
+        instantTime,
+        Option.of(currentMetadata),
+        Option.of(writeStats),
+        previousCommitMetadata,
+        metaClient);
+
+    List<String> classNames = Arrays.stream(validatorClassNames.split(","))
+        .map(String::trim)
+        .filter(s -> !s.isEmpty())
+        .collect(Collectors.toList());
+
+    for (String className : classNames) {
+      try {
+        Class<?> clazz = Class.forName(className);
+        if (!BasePreCommitValidator.class.isAssignableFrom(clazz)) {
+          LOG.warn("Skipping validator {} in HoodieStreamer path β€” it does not extend BasePreCommitValidator. "
+              + "If this is a SparkPreCommitValidator (e.g. SqlQueryEqualityPreCommitValidator), "
+              + "it must be invoked via SparkValidatorUtils in the standard Spark write path instead.", className);
+          continue;
+        }
+        BasePreCommitValidator validator = (BasePreCommitValidator)
+            ReflectionUtils.loadClass(className, new Class<?>[] {TypedProperties.class}, props);
+        LOG.info("Running pre-commit validator: {} for instant: {}", className, instantTime);
+        validator.validateWithMetadata(context);
+        LOG.info("Pre-commit validator {} passed for instant: {}", className, instantTime);
+      } catch (HoodieValidationException e) {
+        LOG.error("Pre-commit validator {} failed for instant: {}", className, instantTime, e);
+        throw e;
+      } catch (Exception e) {
+        LOG.error("Failed to instantiate or run validator: {}", className, e);
+        throw new HoodieValidationException(
+            "Failed to run pre-commit validator: " + className, e);
+      }
+    }
+  }
+
+  /**
+   * Build a pre-commit snapshot of {@link HoodieCommitMetadata} from write statuses and extra metadata.
+   *
+   * <p>This is intentionally a partial/preview object used only for validation β€” it contains
+   * write stats and checkpoint extra-metadata, but omits fields that are not available before the
+   * commit (e.g. schema, operation type). Validators should treat this as a read-only snapshot
+   * of what will be committed, not a fully-constructed commit record.</p>
+   */
+  private static HoodieCommitMetadata buildCommitMetadata(
+      List<WriteStatus> writeStatuses, Map<String, String> extraMetadata) {
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+
+    // Add write stats
+    for (WriteStatus status : writeStatuses) {
+      HoodieWriteStat stat = status.getStat();
+      if (stat != null) {
+        metadata.addWriteStat(stat.getPartitionPath(), stat);
+      }
+    }
+
+    // Add extra metadata (includes checkpoint info like deltastreamer.checkpoint.key)
+    if (extraMetadata != null) {
+      extraMetadata.forEach(metadata::addMetadata);
+    }
+
+    return metadata;
+  }
+
+  /**
+   * Load the previous completed commit metadata from the timeline.
+   */
+  private static Option<HoodieCommitMetadata> loadPreviousCommitMetadata(HoodieTableMetaClient metaClient) {
+    try {
+      HoodieTimeline completedTimeline = metaClient.reloadActiveTimeline()
+          .getWriteTimeline()
+          .filterCompletedInstants();
+      Option<HoodieInstant> lastInstant = completedTimeline.lastInstant();
+      if (lastInstant.isPresent()) {
+        return Option.of(completedTimeline.readCommitMetadata(lastInstant.get()));
+      }
+    } catch (IOException e) {
+      throw new HoodieIOException("Failed to load previous commit metadata", e);
+    }
+    return Option.empty();
+  }
+
+  private SparkStreamerValidatorUtils() {
+    // Utility class
+  }
+}
diff --git a/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkValidationContext.java b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkValidationContext.java
new file mode 100644
index 000000000000..1d46e13aeedb
--- /dev/null
+++ b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/validator/SparkValidationContext.java
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.streamer.validator;
+
+import org.apache.hudi.client.validator.ValidationContext;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+
+import java.util.List;
+
+/**
+ * Spark/HoodieStreamer implementation of {@link ValidationContext}.
+ *
+ * <p>Constructed from data available in {@code StreamSync.writeToSinkAndDoMetaSync()}
+ * before the commit is finalized. Provides validators with access to commit metadata,
+ * write statistics, and previous commit information for streaming offset validation.</p>
+ *
+ * <p>Unlike Flink's implementation, Spark can optionally provide active timeline access
+ * via {@link HoodieTableMetaClient} for richer validation patterns.</p>
+ */
+public class SparkValidationContext implements ValidationContext {
+
+  private final String instantTime;
+  private final Option<HoodieCommitMetadata> commitMetadata;
+  private final Option<List<HoodieWriteStat>> writeStats;
+  private final Option<HoodieCommitMetadata> previousCommitMetadata;
+  private final HoodieTableMetaClient metaClient;
+
+  /**
+   * Create a Spark validation context with full timeline access.
+   *
+   * @param instantTime Current commit instant time
+   * @param commitMetadata Current commit metadata (with extraMetadata including checkpoints)
+   * @param writeStats Write statistics from write operations
+   * @param previousCommitMetadata Metadata from the previous completed commit
+   * @param metaClient Table meta client for timeline access (may be null for testing)
+   */
+  public SparkValidationContext(String instantTime,
+                                Option<HoodieCommitMetadata> commitMetadata,
+                                Option<List<HoodieWriteStat>> writeStats,
+                                Option<HoodieCommitMetadata> previousCommitMetadata,
+                                HoodieTableMetaClient metaClient) {
+    this.instantTime = instantTime;
+    this.commitMetadata = commitMetadata;
+    this.writeStats = writeStats;
+    this.previousCommitMetadata = previousCommitMetadata;
+    this.metaClient = metaClient;
+  }
+
+  /**
+   * Create a Spark validation context without timeline access (for testing).
+   *
+   * @param instantTime Current commit instant time
+   * @param commitMetadata Current commit metadata (with extraMetadata including checkpoints)
+   * @param writeStats Write statistics from write operations
+   * @param previousCommitMetadata Metadata from the previous completed commit
+   */
+  public SparkValidationContext(String instantTime,
+                                Option<HoodieCommitMetadata> commitMetadata,
+                                Option<List<HoodieWriteStat>> writeStats,
+                                Option<HoodieCommitMetadata> previousCommitMetadata) {
+    this(instantTime, commitMetadata, writeStats, previousCommitMetadata, null);
+  }
+
+  @Override
+  public String getInstantTime() {
+    return instantTime;
+  }
+
+  @Override
+  public Option<HoodieCommitMetadata> getCommitMetadata() {
+    return commitMetadata;
+  }
+
+  @Override
+  public Option<List<HoodieWriteStat>> getWriteStats() {
+    return writeStats;
+  }
+
+  /**
+   * Get the active timeline. Available when metaClient is provided.
+   *
+   * @throws UnsupportedOperationException if metaClient was not provided
+   */
+  @Override
+  public HoodieActiveTimeline getActiveTimeline() {
+    if (metaClient == null) {
+      throw new UnsupportedOperationException(
+          "Active timeline is not available without HoodieTableMetaClient.");
+    }
+    return metaClient.getActiveTimeline();
+  }
+
+  /**
+   * Get the previous completed commit instant by querying the timeline.
+   * Returns {@link Option#empty()} if this is the first commit or metaClient is unavailable.
+   */
+  @Override
+  public Option<HoodieInstant> getPreviousCommitInstant() {
+    if (metaClient == null) {
+      return Option.empty();
+    }
+    return metaClient.getActiveTimeline()
+        .getWriteTimeline()
+        .filterCompletedInstants()
+        .lastInstant();
+  }
+
+  @Override
+  public boolean isFirstCommit() {
+    return !previousCommitMetadata.isPresent();
+  }
+
+  @Override
+  public Option<HoodieCommitMetadata> getPreviousCommitMetadata() {
+    return previousCommitMetadata;
+  }
+}
diff --git a/hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkKafkaOffsetValidator.java b/hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkKafkaOffsetValidator.java
new file mode 100644
index 000000000000..d109aa3246f6
--- /dev/null
+++ b/hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkKafkaOffsetValidator.java
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.streamer.validator;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.checkpoint.StreamerCheckpointV1;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodiePreCommitValidatorConfig;
+import org.apache.hudi.exception.HoodieValidationException;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Tests for {@link SparkKafkaOffsetValidator}.
+ */
+public class TestSparkKafkaOffsetValidator {
+
+  // ========== Helper methods ==========
+
+  private static TypedProperties defaultConfig() {
+    TypedProperties props = new TypedProperties();
+    props.setProperty(HoodiePreCommitValidatorConfig.STREAMING_OFFSET_TOLERANCE_PERCENTAGE.key(), "0.0");
+    props.setProperty(HoodiePreCommitValidatorConfig.VALIDATION_FAILURE_POLICY.key(), "FAIL");
+    return props;
+  }
+
+  private static TypedProperties configWithTolerance(double tolerance) {
+    TypedProperties props = defaultConfig();
+    props.setProperty(HoodiePreCommitValidatorConfig.STREAMING_OFFSET_TOLERANCE_PERCENTAGE.key(),
+        String.valueOf(tolerance));
+    return props;
+  }
+
+  private static TypedProperties configWithWarnPolicy() {
+    TypedProperties props = defaultConfig();
+    props.setProperty(HoodiePreCommitValidatorConfig.VALIDATION_FAILURE_POLICY.key(), "WARN_LOG");
+    return props;
+  }
+
+  /**
+   * Build a Spark Kafka checkpoint string.
+   * Format: topic,partition:offset,partition:offset,...
+   */
+  private static String buildSparkKafkaCheckpoint(String topic, int[] partitions, long[] offsets) {
+    StringBuilder sb = new StringBuilder();
+    sb.append(topic);
+    for (int i = 0; i < partitions.length; i++) {
+      sb.append(",").append(partitions[i]).append(":").append(offsets[i]);
+    }
+    return sb.toString();
+  }
+
+  private static HoodieCommitMetadata buildMetadata(String checkpointValue) {
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    if (checkpointValue != null) {
+      metadata.addMetadata(StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1, checkpointValue);
+    }
+    return metadata;
+  }
+
+  private static List<HoodieWriteStat> buildWriteStats(long numInserts, long numUpdates) {
+    HoodieWriteStat stat = new HoodieWriteStat();
+    stat.setNumInserts(numInserts);
+    stat.setNumUpdateWrites(numUpdates);
+    stat.setPartitionPath("partition1");
+    return Collections.singletonList(stat);
+  }
+
+  private static SparkValidationContext buildContext(
+      String instantTime,
+      HoodieCommitMetadata currentMetadata,
+      List<HoodieWriteStat> writeStats,
+      HoodieCommitMetadata previousMetadata) {
+    return new SparkValidationContext(
+        instantTime,
+        Option.of(currentMetadata),
+        Option.of(writeStats),
+        previousMetadata != null ? Option.of(previousMetadata) : Option.empty());
+  }
+
+  // ========== Tests ==========
+
+  @Test
+  public void testExactMatchPasses() {
+    // Previous: partition 0 at offset 100, partition 1 at offset 200
+    // Current: partition 0 at offset 200, partition 1 at offset 300
+    // Diff = (200-100) + (300-200) = 200. Records written = 200.
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0, 1}, new long[]{100, 200});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0, 1}, new long[]{200, 300});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(200, 0),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testDataLossDetected() {
+    // Diff = 1000 but only 500 records written -> 50% deviation
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{0});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{1000});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(500, 0),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertThrows(HoodieValidationException.class, () -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testWithinTolerancePasses() {
+    // Diff = 1000, records = 950 -> 5% deviation, tolerance = 10%
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{0});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{1000});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(950, 0),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(configWithTolerance(10.0));
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testWarnPolicyDoesNotThrow() {
+    // Data loss but WARN_LOG policy
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{0});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{1000});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(0, 0),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(configWithWarnPolicy());
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testSkipsFirstCommit() {
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{1000});
+
+    // No previous commit
+    SparkValidationContext ctx = new SparkValidationContext(
+        "20260320120000000",
+        Option.of(buildMetadata(currCheckpoint)),
+        Option.of(buildWriteStats(500, 0)),
+        Option.empty());
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testSkipsWhenNoCheckpointKey() {
+    // Current metadata has no checkpoint key
+    HoodieCommitMetadata currentMeta = new HoodieCommitMetadata();
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{100});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        currentMeta,
+        buildWriteStats(500, 0),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testMultiPartitionValidation() {
+    // 4 partitions, each advancing by 250 = total diff 1000
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events",
+        new int[]{0, 1, 2, 3}, new long[]{0, 0, 0, 0});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events",
+        new int[]{0, 1, 2, 3}, new long[]{250, 250, 250, 250});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(800, 200),  // 800 inserts + 200 updates = 1000
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testEmptyCommitSkipsValidation() {
+    // Both offsets same and no records written
+    String checkpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{100});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(checkpoint),
+        buildWriteStats(0, 0),
+        buildMetadata(checkpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testPreviousCheckpointMissingSkipsValidation() {
+    // Previous metadata exists but has no checkpoint key
+    HoodieCommitMetadata prevMeta = new HoodieCommitMetadata();
+
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{1000});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(500, 0),
+        prevMeta);
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testOvercountingDetected() {
+    // More records written than offset diff
+    // Diff = 100, records = 200 -> |100-200|/100 = 100% deviation
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{0});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{100});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(200, 0),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertThrows(HoodieValidationException.class, () -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testExactToleranceBoundaryPasses() {
+    // Diff = 1000, records = 900 -> 10% deviation, tolerance = 10%
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{0});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{1000});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(900, 0),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(configWithTolerance(10.0));
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testJustOverToleranceFails() {
+    // Diff = 1000, records = 899 -> 10.1% deviation, tolerance = 10%
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{0});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{1000});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(899, 0),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(configWithTolerance(10.0));
+    assertThrows(HoodieValidationException.class, () -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testOnlyInsertsNoUpdates() {
+    // Pure insert workload
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0, 1}, new long[]{0, 0});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0, 1}, new long[]{500, 500});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(1000, 0),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+
+  @Test
+  public void testUpdatesCountedInRecordTotal() {
+    // Diff = 1000. 600 inserts + 400 updates = 1000 total
+    String prevCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{0});
+    String currCheckpoint = buildSparkKafkaCheckpoint("events", new int[]{0}, new long[]{1000});
+
+    SparkValidationContext ctx = buildContext("20260320120000000",
+        buildMetadata(currCheckpoint),
+        buildWriteStats(600, 400),
+        buildMetadata(prevCheckpoint));
+
+    SparkKafkaOffsetValidator validator = new SparkKafkaOffsetValidator(defaultConfig());
+    assertDoesNotThrow(() -> validator.validateWithMetadata(ctx));
+  }
+}
diff --git a/hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkStreamerValidatorUtils.java b/hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkStreamerValidatorUtils.java
new file mode 100644
index 000000000000..69d5d228dab6
--- /dev/null
+++ b/hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkStreamerValidatorUtils.java
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.streamer.validator;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.common.table.checkpoint.StreamerCheckpointV1;
+import org.apache.hudi.common.table.checkpoint.StreamerCheckpointV2;
+import org.apache.hudi.common.testutils.HoodieTestTable;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodiePreCommitValidatorConfig;
+import org.apache.hudi.exception.HoodieValidationException;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link SparkStreamerValidatorUtils}.
+ *
+ * <p>Tests cover orchestration logic (class loading, config passing, error handling)
+ * as well as end-to-end offset validation using a two-commit timeline to verify
+ * the real comparison path is exercised.</p>
+ */
+public class TestSparkStreamerValidatorUtils {
+
+  @TempDir
+  Path tempDir;
+
+  private static TypedProperties propsWithValidator(String validatorClassName) {
+    TypedProperties props = new TypedProperties();
+    props.setProperty(HoodiePreCommitValidatorConfig.VALIDATOR_CLASS_NAMES.key(), validatorClassName);
+    props.setProperty(HoodiePreCommitValidatorConfig.STREAMING_OFFSET_TOLERANCE_PERCENTAGE.key(), "0.0");
+    props.setProperty(HoodiePreCommitValidatorConfig.VALIDATION_FAILURE_POLICY.key(), "FAIL");
+    return props;
+  }
+
+  private static WriteStatus buildWriteStatus(String partitionPath, long numInserts, long numUpdates) {
+    HoodieWriteStat stat = new HoodieWriteStat();
+    stat.setPartitionPath(partitionPath);
+    stat.setNumInserts(numInserts);
+    stat.setNumUpdateWrites(numUpdates);
+
+    WriteStatus ws = new WriteStatus(false, 0.0);
+    ws.setStat(stat);
+    return ws;
+  }
+
+  private HoodieTableMetaClient createMetaClient() throws IOException {
+    return HoodieTestUtils.init(tempDir.toAbsolutePath().toString());
+  }
+
+  private HoodieTableMetaClient createMetaClient(HoodieTableVersion version) throws IOException {
+    return HoodieTestUtils.init(tempDir.toAbsolutePath().toString(), HoodieTableType.COPY_ON_WRITE, version);
+  }
+
+  // ========== Tests ==========
+
+  @Test
+  public void testNoValidatorsConfigured() throws IOException {
+    TypedProperties props = new TypedProperties();
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 100, 0));
+
+    assertDoesNotThrow(() -> SparkStreamerValidatorUtils.runValidators(
+        props, "20260320120000000", writeStatuses,
+        new HashMap<>(), createMetaClient()));
+  }
+
+  @Test
+  public void testEmptyValidatorString() throws IOException {
+    TypedProperties props = new TypedProperties();
+    props.setProperty(HoodiePreCommitValidatorConfig.VALIDATOR_CLASS_NAMES.key(), "");
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 100, 0));
+
+    assertDoesNotThrow(() -> SparkStreamerValidatorUtils.runValidators(
+        props, "20260320120000000", writeStatuses,
+        new HashMap<>(), createMetaClient()));
+  }
+
+  @Test
+  public void testValidValidatorFirstCommitPasses() throws IOException {
+    TypedProperties props = propsWithValidator(
+        "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator");
+
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 100, 0));
+    Map<String, String> extraMeta = new HashMap<>();
+    extraMeta.put(StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1, "events,0:100");
+
+    // First commit (no previous metadata on timeline) β€” validator should skip and pass
+    assertDoesNotThrow(() -> SparkStreamerValidatorUtils.runValidators(
+        props, "20260320120000000", writeStatuses, extraMeta, createMetaClient()));
+  }
+
+  @Test
+  public void testInvalidValidatorClassThrows() throws IOException {
+    TypedProperties props = propsWithValidator("com.nonexistent.FakeValidator");
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 100, 0));
+
+    assertThrows(HoodieValidationException.class,
+        () -> SparkStreamerValidatorUtils.runValidators(
+            props, "20260320120000000", writeStatuses, new HashMap<>(), createMetaClient()));
+  }
+
+  @Test
+  public void testMultipleValidators() throws IOException {
+    TypedProperties props = propsWithValidator(
+        "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator,"
+            + "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator");
+
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 100, 0));
+    Map<String, String> extraMeta = new HashMap<>();
+    extraMeta.put(StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1, "events,0:100");
+
+    assertDoesNotThrow(() -> SparkStreamerValidatorUtils.runValidators(
+        props, "20260320120000000", writeStatuses, extraMeta, createMetaClient()));
+  }
+
+  @Test
+  public void testValidatorWithWhitespaceInClassNames() throws IOException {
+    TypedProperties props = propsWithValidator(
+        "  org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator  , ");
+
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 100, 0));
+
+    assertDoesNotThrow(() -> SparkStreamerValidatorUtils.runValidators(
+        props, "20260320120000000", writeStatuses, new HashMap<>(), createMetaClient()));
+  }
+
+  @Test
+  public void testNullExtraMetadataHandled() throws IOException {
+    TypedProperties props = propsWithValidator(
+        "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator");
+
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 100, 0));
+
+    assertDoesNotThrow(() -> SparkStreamerValidatorUtils.runValidators(
+        props, "20260320120000000", writeStatuses, null, createMetaClient()));
+  }
+
+  @Test
+  public void testMultipleWriteStatusesAggregated() throws IOException {
+    TypedProperties props = propsWithValidator(
+        "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator");
+
+    List<WriteStatus> writeStatuses = new ArrayList<>();
+    writeStatuses.add(buildWriteStatus("p1", 60, 0));
+    writeStatuses.add(buildWriteStatus("p2", 40, 0));
+
+    Map<String, String> extraMeta = new HashMap<>();
+    extraMeta.put(StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1, "events,0:100");
+
+    assertDoesNotThrow(() -> SparkStreamerValidatorUtils.runValidators(
+        props, "20260320120000000", writeStatuses, extraMeta, createMetaClient()));
+  }
+
+  @Test
+  public void testEmptyWriteStatuses() throws IOException {
+    TypedProperties props = propsWithValidator(
+        "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator");
+
+    List<WriteStatus> writeStatuses = Collections.emptyList();
+    Map<String, String> extraMeta = new HashMap<>();
+    extraMeta.put(StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1, "events,0:100");
+
+    assertDoesNotThrow(() -> SparkStreamerValidatorUtils.runValidators(
+        props, "20260320120000000", writeStatuses, extraMeta, createMetaClient()));
+  }
+
+  @Test
+  public void testValidationExceptionPreservedAcrossValidators() throws IOException {
+    TypedProperties props = propsWithValidator(
+        "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator,"
+            + "com.nonexistent.FakeValidator");
+
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 100, 0));
+
+    HoodieValidationException ex = assertThrows(HoodieValidationException.class,
+        () -> SparkStreamerValidatorUtils.runValidators(
+            props, "20260320120000000", writeStatuses, new HashMap<>(), createMetaClient()));
+    assertTrue(ex.getMessage().contains("FakeValidator"));
+  }
+
+  @ParameterizedTest
+  @ValueSource(strings = {
+      StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1,
+      StreamerCheckpointV2.STREAMER_CHECKPOINT_KEY_V2
+  })
+  public void testSecondCommitMatchingOffsetsPasses(String checkpointKey) throws Exception {
+    TypedProperties props = propsWithValidator(
+        "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator");
+
+    // Create table with a previous committed instant: offset 0 -> 500
+    HoodieTableMetaClient metaClient = createMetaClient();
+    HoodieCommitMetadata prevMeta = new HoodieCommitMetadata();
+    prevMeta.addMetadata(checkpointKey, "events,0:500");
+    HoodieTestTable.of(metaClient).addCommit("20260320110000000", Option.of(prevMeta));
+
+    // Second commit: offset 500 -> 600, 100 records written β€” matches diff exactly
+    Map<String, String> extraMeta = new HashMap<>();
+    extraMeta.put(checkpointKey, "events,0:600");
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 100, 0));
+
+    assertDoesNotThrow(() -> SparkStreamerValidatorUtils.runValidators(
+        props, "20260320120000000", writeStatuses, extraMeta, metaClient));
+  }
+
+  @ParameterizedTest
+  @ValueSource(strings = {
+      StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1,
+      StreamerCheckpointV2.STREAMER_CHECKPOINT_KEY_V2
+  })
+  public void testSecondCommitDataLossDetected(String checkpointKey) throws Exception {
+    TypedProperties props = propsWithValidator(
+        "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator");
+
+    // Create table with a previous committed instant: offset 0 -> 1000
+    HoodieTableMetaClient metaClient = createMetaClient();
+    HoodieCommitMetadata prevMeta = new HoodieCommitMetadata();
+    prevMeta.addMetadata(checkpointKey, "events,0:1000");
+    HoodieTestTable.of(metaClient).addCommit("20260320110000000", Option.of(prevMeta));
+
+    // Second commit: offset 1000 -> 2000 (diff=1000) but only 500 records written β€” data loss
+    Map<String, String> extraMeta = new HashMap<>();
+    extraMeta.put(checkpointKey, "events,0:2000");
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 500, 0));
+
+    assertThrows(HoodieValidationException.class,
+        () -> SparkStreamerValidatorUtils.runValidators(
+            props, "20260320120000000", writeStatuses, extraMeta, metaClient));
+  }
+
+  @Test
+  public void testV2CheckpointKeyOnTableVersionEightFires() throws Exception {
+    // Verifies the validator actually fires on a writeTableVersion=8 table that uses the
+    // V2 checkpoint key β€” i.e. the auto-resolution in StreamingOffsetValidator picks up V2
+    // and runs the comparison instead of silently skipping.
+    TypedProperties props = propsWithValidator(
+        "org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator");
+
+    HoodieTableMetaClient metaClient = createMetaClient(HoodieTableVersion.EIGHT);
+    HoodieCommitMetadata prevMeta = new HoodieCommitMetadata();
+    prevMeta.addMetadata(StreamerCheckpointV2.STREAMER_CHECKPOINT_KEY_V2, "events,0:1000");
+    HoodieTestTable.of(metaClient).addCommit("20260320110000000", Option.of(prevMeta));
+
+    // Offset diff = 1000 but only 200 records written β€” must fail
+    Map<String, String> extraMeta = new HashMap<>();
+    extraMeta.put(StreamerCheckpointV2.STREAMER_CHECKPOINT_KEY_V2, "events,0:2000");
+    List<WriteStatus> writeStatuses = Collections.singletonList(buildWriteStatus("p1", 200, 0));
+
+    assertThrows(HoodieValidationException.class,
+        () -> SparkStreamerValidatorUtils.runValidators(
+            props, "20260320120000000", writeStatuses, extraMeta, metaClient));
+  }
+}
diff --git a/hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkValidationContext.java b/hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkValidationContext.java
new file mode 100644
index 000000000000..7f94262e98c4
--- /dev/null
+++ b/hudi-utilities/src/test/java/org/apache/hudi/utilities/streamer/validator/TestSparkValidationContext.java
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities.streamer.validator;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.checkpoint.StreamerCheckpointV1;
+import org.apache.hudi.common.util.Option;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests for {@link SparkValidationContext}.
+ */
+public class TestSparkValidationContext {
+
+  private static HoodieWriteStat buildStat(long inserts, long updates) {
+    HoodieWriteStat stat = new HoodieWriteStat();
+    stat.setNumInserts(inserts);
+    stat.setNumUpdateWrites(updates);
+    stat.setPartitionPath("partition1");
+    return stat;
+  }
+
+  @Test
+  public void testBasicProperties() {
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    metadata.addMetadata("key1", "value1");
+    List<HoodieWriteStat> writeStats = Collections.singletonList(buildStat(100, 50));
+
+    SparkValidationContext ctx = new SparkValidationContext(
+        "20260320120000000",
+        Option.of(metadata),
+        Option.of(writeStats),
+        Option.empty());
+
+    assertEquals("20260320120000000", ctx.getInstantTime());
+    assertTrue(ctx.getCommitMetadata().isPresent());
+    assertTrue(ctx.getWriteStats().isPresent());
+    assertEquals(1, ctx.getWriteStats().get().size());
+  }
+
+  @Test
+  public void testRecordCounting() {
+    List<HoodieWriteStat> writeStats = Arrays.asList(
+        buildStat(100, 50),   // partition1: 100 inserts, 50 updates
+        buildStat(200, 30));  // partition2: 200 inserts, 30 updates
+
+    SparkValidationContext ctx = new SparkValidationContext(
+        "20260320120000000",
+        Option.of(new HoodieCommitMetadata()),
+        Option.of(writeStats),
+        Option.empty());
+
+    assertEquals(300, ctx.getTotalInsertRecordsWritten());
+    assertEquals(80, ctx.getTotalUpdateRecordsWritten());
+    assertEquals(380, ctx.getTotalRecordsWritten());
+  }
+
+  @Test
+  public void testFirstCommitDetection() {
+    // No previous commit metadata -> first commit
+    SparkValidationContext ctx = new SparkValidationContext(
+        "20260320120000000",
+        Option.of(new HoodieCommitMetadata()),
+        Option.of(Collections.emptyList()),
+        Option.empty());
+
+    assertTrue(ctx.isFirstCommit());
+  }
+
+  @Test
+  public void testNotFirstCommitWhenPreviousExists() {
+    HoodieCommitMetadata prevMeta = new HoodieCommitMetadata();
+
+    SparkValidationContext ctx = new SparkValidationContext(
+        "20260320120000000",
+        Option.of(new HoodieCommitMetadata()),
+        Option.of(Collections.emptyList()),
+        Option.of(prevMeta));
+
+    assertFalse(ctx.isFirstCommit());
+  }
+
+  @Test
+  public void testExtraMetadataAccess() {
+    HoodieCommitMetadata metadata = new HoodieCommitMetadata();
+    metadata.addMetadata(StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1, "events,0:1000");
+    metadata.addMetadata("custom.key", "custom_value");
+
+    SparkValidationContext ctx = new SparkValidationContext(
+        "20260320120000000",
+        Option.of(metadata),
+        Option.of(Collections.emptyList()),
+        Option.empty());
+
+    assertEquals("events,0:1000",
+        ctx.getExtraMetadata(StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1).get());
+    assertEquals("custom_value", ctx.getExtraMetadata("custom.key").get());
+    assertFalse(ctx.getExtraMetadata("nonexistent.key").isPresent());
+  }
+
+  @Test
+  public void testPreviousCommitMetadataAccess() {
+    HoodieCommitMetadata prevMeta = new HoodieCommitMetadata();
+    prevMeta.addMetadata(StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1, "events,0:500");
+
+    SparkValidationContext ctx = new SparkValidationContext(
+        "20260320120000000",
+        Option.of(new HoodieCommitMetadata()),
+        Option.of(Collections.emptyList()),
+        Option.of(prevMeta));
+
+    assertTrue(ctx.getPreviousCommitMetadata().isPresent());
+    assertEquals("events,0:500",
+        ctx.getPreviousCommitMetadata().get().getMetadata(StreamerCheckpointV1.STREAMER_CHECKPOINT_KEY_V1));
+  }
+
+  @Test
+  public void testEmptyWriteStats() {
+    SparkValidationContext ctx = new SparkValidationContext(
+        "20260320120000000",
+        Option.of(new HoodieCommitMetadata()),
+        Option.empty(),
+        Option.empty());
+
+    assertEquals(0, ctx.getTotalRecordsWritten());
+    assertEquals(0, ctx.getTotalInsertRecordsWritten());
+    assertEquals(0, ctx.getTotalUpdateRecordsWritten());
+  }
+}


(hudi) 12/16: chore: migrate the flink ITs run to flink2.1 (#18717)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit a3272bd3293a7b01a66bb1efd602d6d1ed430004
Author: Danny Chan <yu...@gmail.com>
AuthorDate: Tue May 12 17:09:39 2026 +0800

    chore: migrate the flink ITs run to flink2.1 (#18717)
    
    * chore: migrate the flink ITs to run with flink2.1
    
    * fix compile errors
    
    * fix test failures
    
    * fix test failure
    
    * address review comments
    
    (cherry picked from commit b6bc1658e8f2862583c22425eaa0665ae100f5a3)
---
 .github/workflows/bot.yml                          | 30 ++++----
 azure-pipelines-20230430.yml                       |  4 +-
 .../hudi/util/TestHoodieSchemaConverter.java       | 23 +++---
 .../hudi/sink/bootstrap/RLIBootstrapOperator.java  | 27 ++++---
 .../sink/bootstrap/TestRLIBootstrapOperator.java   | 80 +++++++++++++++++++++
 .../ITTestVariantCrossEngineCompatibility.java     | 31 ++++----
 .../hudi/adapter/DataTypeAdapterTestUtils.java     | 28 ++++++++
 .../hudi/adapter/DataTypeAdapterTestUtils.java     | 28 ++++++++
 .../hudi/adapter/DataTypeAdapterTestUtils.java     | 28 ++++++++
 .../hudi/adapter/DataTypeAdapterTestUtils.java     | 28 ++++++++
 .../runtime/kryo/KryoSerializerSnapshot.java       |  3 +
 .../hudi/adapter/DataTypeAdapterTestUtils.java     | 28 ++++++++
 .../table/format/cow/ParquetSplitReaderUtil.java   | 83 ++++++++++++++++++++++
 .../runtime/kryo/KryoSerializerSnapshot.java       |  3 +
 .../hudi/adapter/DataTypeAdapterTestUtils.java     | 32 +++++++++
 pom.xml                                            | 14 ++--
 16 files changed, 404 insertions(+), 66 deletions(-)

diff --git a/.github/workflows/bot.yml b/.github/workflows/bot.yml
index 1e50c64121cb..06468d0c4fa2 100644
--- a/.github/workflows/bot.yml
+++ b/.github/workflows/bot.yml
@@ -67,7 +67,7 @@ jobs:
         include:
           - scalaProfile: "scala-2.12"
             sparkProfile: "spark3.5"
-            flinkProfile: "flink1.18"
+            flinkProfile: "flink2.1"
 
     steps:
       - uses: actions/checkout@v5
@@ -126,7 +126,7 @@ jobs:
         include:
           - scalaProfile: "scala-2.12"
             sparkProfile: "spark3.5"
-            flinkProfile: "flink1.18"
+            flinkProfile: "flink2.1"
 
     steps:
       - uses: actions/checkout@v5
@@ -178,7 +178,7 @@ jobs:
         include:
           - scalaProfile: "scala-2.12"
             sparkProfile: "spark3.5"
-            flinkProfile: "flink1.18"
+            flinkProfile: "flink2.1"
 
     env:
       UT_MODULES: >-
@@ -563,7 +563,7 @@ jobs:
         include:
           - scalaProfile: "scala-2.12"
             sparkProfile: "spark3.5"
-            flinkProfile: "flink1.20"
+            flinkProfile: "flink2.1"
 
     steps:
       - uses: actions/checkout@v5
@@ -938,7 +938,7 @@ jobs:
           FLINK_PROFILE: ${{ matrix.flinkProfile }}
           FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
           FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
-        if: ${{ endsWith(env.FLINK_PROFILE, '1.20') }}
+        if: ${{ endsWith(env.FLINK_PROFILE, '2.1') }}
         run: |
           mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
           mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" $FLINK_IT_FILTER1 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
@@ -948,9 +948,9 @@ jobs:
     strategy:
       matrix:
         include:
-          - flinkProfile: "flink1.20"
+          - flinkProfile: "flink2.1"
             flinkAvroVersion: "1.11.4"
-            flinkParquetVersion: '1.13.1'
+            flinkParquetVersion: '1.15.2'
     steps:
       - uses: actions/checkout@v5
       - name: Set up JDK 11
@@ -974,7 +974,7 @@ jobs:
           FLINK_PROFILE: ${{ matrix.flinkProfile }}
           FLINK_AVRO_VERSION: ${{ matrix.flinkAvroVersion }}
           FLINK_PARQUET_VERSION: ${{ matrix.flinkParquetVersion }}
-        if: ${{ endsWith(env.FLINK_PROFILE, '1.20') }}
+        if: ${{ endsWith(env.FLINK_PROFILE, '2.1') }}
         run: |
           mvn clean install -T 2 -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -pl hudi-flink-datasource/hudi-flink -am -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" -DskipTests=true $MVN_ARGS
           mvn verify -Pintegration-tests -D"$SCALA_PROFILE" -D"$FLINK_PROFILE" -Davro.version="$FLINK_AVRO_VERSION" -Dparquet.version="$FLINK_PARQUET_VERSION" $FLINK_IT_FILTER2 -pl hudi-flink-datasource/hudi-flink $MVN_ARGS
@@ -985,19 +985,19 @@ jobs:
       matrix:
         include:
           - scalaProfile: 'scala-2.13'
-            flinkProfile: 'flink1.20'
+            flinkProfile: 'flink2.1'
             sparkProfile: 'spark3.5'
             sparkRuntime: 'spark3.5.0'
           - scalaProfile: 'scala-2.12'
-            flinkProfile: 'flink1.20'
+            flinkProfile: 'flink2.1'
             sparkProfile: 'spark3.5'
             sparkRuntime: 'spark3.5.0'
           - scalaProfile: 'scala-2.13'
-            flinkProfile: 'flink1.20'
+            flinkProfile: 'flink2.1'
             sparkProfile: 'spark4.0'
             sparkRuntime: 'spark4.0.0'
           - scalaProfile: 'scala-2.13'
-            flinkProfile: 'flink1.20'
+            flinkProfile: 'flink2.1'
             sparkProfile: 'spark4.1'
             sparkRuntime: 'spark4.1.1'
 
@@ -1201,7 +1201,7 @@ jobs:
       matrix:
         include:
           - sparkProfile: 'spark3.5'
-            flinkProfile: 'flink1.20'
+            flinkProfile: 'flink2.1'
             sparkArchive: 'spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz'
     steps:
       - uses: actions/checkout@v5
@@ -1297,9 +1297,9 @@ jobs:
       matrix:
         include:
           - scalaProfile: "scala-2.12"
-            flinkProfile: "flink1.20"
+            flinkProfile: "flink2.1"
             flinkAvroVersion: '1.11.4'
-            flinkParquetVersion: '1.13.1'
+            flinkParquetVersion: '1.15.2'
     steps:
       - uses: actions/checkout@v5
       - name: Set up JDK 17
diff --git a/azure-pipelines-20230430.yml b/azure-pipelines-20230430.yml
index 10fab9f34992..3edac6046b1a 100644
--- a/azure-pipelines-20230430.yml
+++ b/azure-pipelines-20230430.yml
@@ -14,7 +14,7 @@
 # limitations under the License.
 
 # NOTE:
-# This config file defines how Azure CI runs tests with Spark 3.5 and Flink 1.18 profiles.
+# This config file defines how Azure CI runs tests with Spark 3.5 and Flink 2.1 profiles.
 # PRs will need to keep in sync with master's version to trigger the CI runs.
 # See scripts/jacoco/README.md for how aggregated code coverage report works
 # across multiple modules.
@@ -101,7 +101,7 @@ parameters:
       - '!packaging/hudi-utilities-slim-bundle'
 
 variables:
-  BUILD_PROFILES: '-Dscala-2.12 -Dspark3.5 -Dflink1.18'
+  BUILD_PROFILES: '-Dscala-2.12 -Dspark3.5 -Dflink2.1'
   PLUGIN_OPTS: '-Dcheckstyle.skip=true -Drat.skip=true -ntp -B -V -Pwarn-log -Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.shade=warn -Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.dependency=warn'
   MVN_OPTS_INSTALL: '-T 3 -Phudi-platform-service -DskipTests $(BUILD_PROFILES) $(PLUGIN_OPTS) -Dmaven.wagon.httpconnectionManager.ttlSeconds=25 -Dmaven.wagon.http.retryHandler.count=5 -Dorg.eclipse.jetty.LEVEL=WARN'
   MVN_OPTS_TEST: '-fae -Pwarn-log $(BUILD_PROFILES) $(PLUGIN_OPTS)'
diff --git a/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java b/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java
index 51b8a15f4c7b..9a4f920b859e 100644
--- a/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java
+++ b/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java
@@ -36,11 +36,12 @@ import org.apache.flink.table.types.logical.RowType;
 import org.apache.flink.table.types.logical.TimestampType;
 import org.apache.flink.table.types.logical.VarCharType;
 import org.apache.flink.table.types.logical.VarBinaryType;
-import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 
 import java.util.Arrays;
 
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.junit.jupiter.api.Assertions.assertInstanceOf;
@@ -693,24 +694,18 @@ public class TestHoodieSchemaConverter {
   }
 
   @Test
-  @Disabled("disabled and reopen the tests for 1.3")
   public void testVariantTypeConversion() {
     // Test direct Variant conversion
     HoodieSchema variantSchema = HoodieSchema.createVariant();
     DataType dataType = HoodieSchemaConverter.convertToDataType(variantSchema);
     assertNotNull(dataType);
 
-    // Verify it's a ROW with metadata and value binary fields
-    RowType rowType = (RowType) dataType.getLogicalType();
-    assertEquals(2, rowType.getFieldCount());
-    assertEquals("metadata", rowType.getFieldNames().get(0));
-    assertEquals("value", rowType.getFieldNames().get(1));
-    assertInstanceOf(VarBinaryType.class, rowType.getTypeAt(0));
-    assertInstanceOf(VarBinaryType.class, rowType.getTypeAt(1));
+    // Verify it's a Variant
+    assertThat("the return type should be variant",
+        dataType.getLogicalType().asSummaryString(), is("VARIANT NOT NULL"));
   }
 
   @Test
-  @Disabled("disabled and reopen the tests for 1.3")
   public void testVariantInRecordConversion() {
     // Test Variant field within a record
     HoodieSchema recordWithVariant = HoodieSchema.createRecord(
@@ -727,11 +722,9 @@ public class TestHoodieSchemaConverter {
     assertEquals(2, result.getFieldCount());
     assertEquals("data", result.getFieldNames().get(1));
 
-    // Verify variant field is a ROW<metadata BYTES, value BYTES>
-    RowType variantRowType = (RowType) result.getTypeAt(1);
-    assertEquals(2, variantRowType.getFieldCount());
-    assertEquals("metadata", variantRowType.getFieldNames().get(0));
-    assertEquals("value", variantRowType.getFieldNames().get(1));
+    // Verify variant field
+    assertThat("the return type should be variant",
+        result.getTypeAt(1).asSummaryString(), is("VARIANT NOT NULL"));
   }
 
   @Test
diff --git a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/RLIBootstrapOperator.java b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/RLIBootstrapOperator.java
index 515b180d3cad..d0cb16a789e5 100644
--- a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/RLIBootstrapOperator.java
+++ b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/bootstrap/RLIBootstrapOperator.java
@@ -24,6 +24,7 @@ import org.apache.hudi.common.data.HoodiePairData;
 import org.apache.hudi.common.function.SerializableFunctionUnchecked;
 import org.apache.hudi.common.model.FileSlice;
 import org.apache.hudi.common.model.HoodieRecordGlobalLocation;
+import org.apache.hudi.common.table.HoodieTableConfig;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
 import org.apache.hudi.configuration.FlinkOptions;
 import org.apache.hudi.metadata.HoodieBackedTableMetadata;
@@ -52,7 +53,7 @@ import java.util.stream.Collectors;
 public class RLIBootstrapOperator
     extends AbstractBootstrapOperator {
 
-  private transient HoodieBackedTableMetadata metadataTable;
+  private transient HoodieBackedTableMetadata tableMetadata;
   private transient long loadedCnt;
 
   public RLIBootstrapOperator(Configuration conf) {
@@ -63,13 +64,13 @@ public class RLIBootstrapOperator
   public void initializeState(StateInitializationContext context) throws Exception {
     loadedCnt = 0;
     HoodieTableMetaClient metaClient = StreamerUtil.createMetaClient(conf);
-    this.metadataTable = (HoodieBackedTableMetadata) metaClient.getTableFormat().getMetadataFactory().create(
+    this.tableMetadata = new HoodieBackedTableMetadata(
         HoodieFlinkEngineContext.DEFAULT,
         metaClient.getStorage(),
         StreamerUtil.metadataConfig(conf),
         conf.get(FlinkOptions.PATH));
     // Load RLI records
-    preLoadRLIRecords();
+    preLoadRLIRecords(metaClient.getTableConfig());
   }
 
   @Override
@@ -82,10 +83,20 @@ public class RLIBootstrapOperator
   //  Utilities
   // -------------------------------------------------------------------------
 
-  private void preLoadRLIRecords() {
+  private void preLoadRLIRecords(HoodieTableConfig tableConfig) {
     int taskID = RuntimeContextUtils.getIndexOfThisSubtask(getRuntimeContext());
     int parallelism = RuntimeContextUtils.getNumberOfParallelSubtasks(getRuntimeContext());
 
+    if (!tableMetadata.enabled()) {
+      if (tableConfig.isMetadataTableAvailable()) {
+        throw new RuntimeException("Can not initialize the table metadata");
+      }
+      log.info("Skip loading RLI records because table metadata is not initialized, taskId = {}", taskID);
+      waitForBootstrapReady(taskID);
+      closeMetadataTable();
+      return;
+    }
+
     log.info("Start loading RLI records from metadata table, taskId = {}, parallelism = {}", taskID, parallelism);
 
     SerializableFunctionUnchecked<List<FileSlice>, List<FileSlice>> fileSlicesFilter = fileSlices -> {
@@ -102,7 +113,7 @@ public class RLIBootstrapOperator
 
     // Each subtask loads buckets assigned to it
     long startTime = System.currentTimeMillis();
-    HoodiePairData<String, HoodieRecordGlobalLocation> rliData = metadataTable.readRecordIndexLocations(fileSlicesFilter);
+    HoodiePairData<String, HoodieRecordGlobalLocation> rliData = tableMetadata.readRecordIndexLocations(fileSlicesFilter);
     rliData.forEach(locationPair -> emitIndexRecord(locationPair.getLeft(), locationPair.getRight()));
     long costMs = System.currentTimeMillis() - startTime;
     log.info("Finish loading RLI records, total records: {}, cost: {} ms, taskId = {}", loadedCnt, costMs, taskID);
@@ -133,13 +144,13 @@ public class RLIBootstrapOperator
   }
 
   private void closeMetadataTable() {
-    if (metadataTable != null) {
+    if (tableMetadata != null) {
       try {
-        metadataTable.close();
+        tableMetadata.close();
       } catch (Exception e) {
         log.warn("Failed to close metadata table", e);
       }
-      metadataTable = null;
+      tableMetadata = null;
     }
   }
 }
diff --git a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/bootstrap/TestRLIBootstrapOperator.java b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/bootstrap/TestRLIBootstrapOperator.java
new file mode 100644
index 000000000000..2e684d0652ae
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/bootstrap/TestRLIBootstrapOperator.java
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.bootstrap;
+
+import org.apache.hudi.client.model.HoodieFlinkInternalRow;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.index.HoodieIndex;
+import org.apache.hudi.metadata.MetadataPartitionType;
+import org.apache.hudi.util.StreamerUtil;
+import org.apache.hudi.utils.TestConfigurations;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.File;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Tests for {@link RLIBootstrapOperator}.
+ */
+public class TestRLIBootstrapOperator {
+
+  @TempDir
+  File tempFile;
+
+  @Test
+  void testSkipPreloadForFreshTableWithoutMetadataTable() throws Exception {
+    Configuration conf = getRLIConf();
+    StreamerUtil.initTableIfNotExists(conf);
+
+    try (OneInputStreamOperatorTestHarness<HoodieFlinkInternalRow, HoodieFlinkInternalRow> harness =
+             new OneInputStreamOperatorTestHarness<>(new RLIBootstrapOperator(conf), 1, 1, 0)) {
+      harness.open();
+
+      assertEquals(0, harness.getOutput().size());
+    }
+  }
+
+  @Test
+  void testFailFastWhenMetadataTableIsMarkedAvailableButCannotBeLoaded() throws Exception {
+    Configuration conf = getRLIConf();
+    HoodieTableMetaClient metaClient = StreamerUtil.initTableIfNotExists(conf);
+    metaClient.getTableConfig().setMetadataPartitionState(metaClient, MetadataPartitionType.FILES.getPartitionPath(), true);
+
+    try (OneInputStreamOperatorTestHarness<HoodieFlinkInternalRow, HoodieFlinkInternalRow> harness =
+             new OneInputStreamOperatorTestHarness<>(new RLIBootstrapOperator(conf), 1, 1, 0)) {
+      RuntimeException error = assertThrows(RuntimeException.class, harness::open);
+
+      assertEquals("Can not initialize the table metadata", error.getMessage());
+    }
+  }
+
+  private Configuration getRLIConf() {
+    Configuration conf = TestConfigurations.getDefaultConf(tempFile.getAbsolutePath());
+    conf.set(FlinkOptions.METADATA_ENABLED, true);
+    conf.set(FlinkOptions.INDEX_TYPE, HoodieIndex.IndexType.GLOBAL_RECORD_LEVEL_INDEX.name());
+    return conf;
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java
index 03305d478ecd..de6e3835d120 100644
--- a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java
+++ b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java
@@ -18,6 +18,8 @@
 
 package org.apache.hudi.table;
 
+import org.apache.hudi.adapter.DataTypeAdapter;
+import org.apache.hudi.adapter.DataTypeAdapterTestUtils;
 import org.apache.hudi.common.testutils.HoodieTestUtils;
 import org.apache.hudi.common.util.StringUtils;
 import org.apache.hudi.utils.FlinkMiniCluster;
@@ -27,7 +29,6 @@ import org.apache.flink.table.api.TableEnvironment;
 import org.apache.flink.table.api.TableResult;
 import org.apache.flink.types.Row;
 import org.apache.flink.util.CollectionUtil;
-import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.junit.jupiter.api.io.TempDir;
@@ -51,19 +52,16 @@ public class ITTestVariantCrossEngineCompatibility {
 
   /**
    * Helper method to verify that Flink can read Spark 4.0 Variant tables.
-   * Variant data is represented as ROW<metadata BYTES, value BYTES> in Flink.
    */
   private void verifyFlinkCanReadSparkVariantTable(String tablePath, String tableType, String testDescription) throws Exception {
     TableEnvironment tableEnv = TestTableEnvs.getBatchTableEnv();
 
     // Create a Hudi table pointing to the Spark-written data
-    // In Flink, Variant is represented as ROW<metadata BYTES, value BYTES>
-    // NOTE: value is a reserved keyword
     String createTableDdl = String.format(
         "CREATE TABLE variant_table ("
             + "  id INT,"
             + "  name STRING,"
-            + "  v ROW<metadata BYTES, `value` BYTES>,"
+            + "  v VARIANT,"
             + "  ts BIGINT,"
             + "  PRIMARY KEY (id) NOT ENFORCED"
             + ") WITH ("
@@ -87,35 +85,32 @@ public class ITTestVariantCrossEngineCompatibility {
     assertEquals("row1", row.getField(1), "Second column should be name=row1");
     assertEquals(1000L, row.getField(3), "Fourth column should be ts=1000");
 
-    // Verify the variant column is readable as a ROW with binary fields
-    Row variantRow = (Row) row.getField(2);
-    assertNotNull(variantRow, "Variant column should not be null");
-
-    byte[] metadataBytes = (byte[]) variantRow.getField(0);
-    byte[] valueBytes = (byte[]) variantRow.getField(1);
+    // Verify the variant column is readable as a native Flink Variant.
+    Object variantObject = row.getField(2);
+    assertNotNull(variantObject, "Variant column should not be null");
+    DataTypeAdapterTestUtils.assertAsBinaryVariant(variantObject);
 
     // Expected byte values from Spark 4.0 Variant representation: {"updated": true, "new_field": 123}
     byte[] expectedValueBytes = new byte[]{0x02, 0x02, 0x01, 0x00, 0x01, 0x00, 0x03, 0x04, 0x0C, 0x7B};
     byte[] expectedMetadataBytes = new byte[]{0x01, 0x02, 0x00, 0x07, 0x10, 0x75, 0x70, 0x64, 0x61,
         0x74, 0x65, 0x64, 0x6E, 0x65, 0x77, 0x5F, 0x66, 0x69, 0x65, 0x6C, 0x64};
 
-    assertArrayEquals(expectedValueBytes, valueBytes,
+    assertArrayEquals(expectedValueBytes, DataTypeAdapter.getVariantValue(variantObject),
         String.format("Variant value bytes mismatch (%s). Expected: %s, Got: %s",
             testDescription,
             Arrays.toString(StringUtils.encodeHex(expectedValueBytes)),
-            Arrays.toString(StringUtils.encodeHex(valueBytes))));
+            Arrays.toString(StringUtils.encodeHex(DataTypeAdapter.getVariantValue(variantObject)))));
 
-    assertArrayEquals(expectedMetadataBytes, metadataBytes,
+    assertArrayEquals(expectedMetadataBytes, DataTypeAdapter.getVariantMetadata(variantObject),
         String.format("Variant metadata bytes mismatch (%s). Expected: %s, Got: %s",
             testDescription,
             Arrays.toString(StringUtils.encodeHex(expectedMetadataBytes)),
-            Arrays.toString(StringUtils.encodeHex(metadataBytes))));
+            Arrays.toString(StringUtils.encodeHex(DataTypeAdapter.getVariantMetadata(variantObject)))));
 
     tableEnv.executeSql("DROP TABLE variant_table");
   }
 
   @Test
-  @Disabled("disabled and reopen the tests for 1.3")
   public void testFlinkReadSparkVariantCOWTable() throws Exception {
     // Test that Flink can read a COW table with Variant data written by Spark 4.0
     Path cowTargetDir = tempDir.resolve("cow");
@@ -125,7 +120,6 @@ public class ITTestVariantCrossEngineCompatibility {
   }
 
   @Test
-  @Disabled("disabled and reopen the tests for 1.3")
   public void testFlinkReadSparkVariantMORTableWithAvro() throws Exception {
     // Test that Flink can read a MOR table with AVRO record type and Variant data written by Spark 4.0
     Path morAvroTargetDir = tempDir.resolve("mor_avro");
@@ -135,7 +129,6 @@ public class ITTestVariantCrossEngineCompatibility {
   }
 
   @Test
-  @Disabled("disabled and reopen the tests for 1.3")
   public void testFlinkReadSparkVariantMORTableWithSpark() throws Exception {
     // Test that Flink can read a MOR table with SPARK record type and Variant data written by Spark 4.0
     Path morSparkTargetDir = tempDir.resolve("mor_spark");
@@ -143,4 +136,4 @@ public class ITTestVariantCrossEngineCompatibility {
     String morSparkPath = morSparkTargetDir.resolve("variant_mor_spark").toString();
     verifyFlinkCanReadSparkVariantTable(morSparkPath, "MERGE_ON_READ", "MOR table with SPARK record type");
   }
-}
\ No newline at end of file
+}
diff --git a/hudi-flink-datasource/hudi-flink1.17.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java b/hudi-flink-datasource/hudi-flink1.17.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
new file mode 100644
index 000000000000..ae2e4107d6ea
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.17.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+/**
+ * Adapter utils.
+ */
+public class DataTypeAdapterTestUtils {
+  public static void assertAsBinaryVariant(Object variantObject) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java b/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
new file mode 100644
index 000000000000..ae2e4107d6ea
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+/**
+ * Adapter utils.
+ */
+public class DataTypeAdapterTestUtils {
+  public static void assertAsBinaryVariant(Object variantObject) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink1.19.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java b/hudi-flink-datasource/hudi-flink1.19.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
new file mode 100644
index 000000000000..ae2e4107d6ea
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.19.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+/**
+ * Adapter utils.
+ */
+public class DataTypeAdapterTestUtils {
+  public static void assertAsBinaryVariant(Object variantObject) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink1.20.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java b/hudi-flink-datasource/hudi-flink1.20.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
new file mode 100644
index 000000000000..ae2e4107d6ea
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.20.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+/**
+ * Adapter utils.
+ */
+public class DataTypeAdapterTestUtils {
+  public static void assertAsBinaryVariant(Object variantObject) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink2.0.x/src/test/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializerSnapshot.java b/hudi-flink-datasource/hudi-flink2.0.x/src/test/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializerSnapshot.java
index 1a256ffe7bba..aac9a4651aff 100644
--- a/hudi-flink-datasource/hudi-flink2.0.x/src/test/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializerSnapshot.java
+++ b/hudi-flink-datasource/hudi-flink2.0.x/src/test/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializerSnapshot.java
@@ -34,6 +34,9 @@ public class KryoSerializerSnapshot<T> implements TypeSerializerSnapshot<T> {
 
   private Class<T> type;
 
+  public KryoSerializerSnapshot() {
+  }
+
   public KryoSerializerSnapshot(Class<T> type) {
     this.type = type;
   }
diff --git a/hudi-flink-datasource/hudi-flink2.0.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java b/hudi-flink-datasource/hudi-flink2.0.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
new file mode 100644
index 000000000000..ae2e4107d6ea
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink2.0.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+/**
+ * Adapter utils.
+ */
+public class DataTypeAdapterTestUtils {
+  public static void assertAsBinaryVariant(Object variantObject) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/table/format/cow/ParquetSplitReaderUtil.java b/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/table/format/cow/ParquetSplitReaderUtil.java
index bb5d0c55b81d..3e16417ba541 100644
--- a/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/table/format/cow/ParquetSplitReaderUtil.java
+++ b/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/table/format/cow/ParquetSplitReaderUtil.java
@@ -18,6 +18,7 @@
 
 package org.apache.hudi.table.format.cow;
 
+import org.apache.hudi.common.schema.HoodieSchema;
 import org.apache.hudi.common.util.ValidationUtils;
 import org.apache.hudi.table.format.cow.vector.HeapArrayGroupColumnVector;
 import org.apache.hudi.table.format.cow.vector.HeapArrayVector;
@@ -320,6 +321,17 @@ public class ParquetSplitReaderUtil {
         } else {
           throw new UnsupportedOperationException("Unsupported create row with default value.");
         }
+      case VARIANT:
+        HeapRowColumnVector variantVector = new HeapRowColumnVector(
+            batchSize,
+            new HeapBytesVector(batchSize),
+            new HeapBytesVector(batchSize));
+        if (value == null) {
+          variantVector.fillWithNulls();
+          return variantVector;
+        } else {
+          throw new UnsupportedOperationException("Unsupported create variant with default value.");
+        }
       default:
         throw new UnsupportedOperationException("Unsupported type: " + type);
     }
@@ -498,6 +510,20 @@ public class ParquetSplitReaderUtil {
           }
         }
         return new RowColumnReader(fieldReaders);
+      case VARIANT:
+        ColumnDescriptor valueDescriptor = getVariantColumnDescriptor(
+            physicalType,
+            descriptors,
+            depth,
+            HoodieSchema.Variant.VARIANT_VALUE_FIELD);
+        ColumnDescriptor metadataDescriptor = getVariantColumnDescriptor(
+            physicalType,
+            descriptors,
+            depth,
+            HoodieSchema.Variant.VARIANT_METADATA_FIELD);
+        return new RowColumnReader(Arrays.asList(
+            new BytesColumnReader(valueDescriptor, pages.getPageReader(valueDescriptor)),
+            new BytesColumnReader(metadataDescriptor, pages.getPageReader(metadataDescriptor))));
       default:
         throw new UnsupportedOperationException(fieldType + " is not supported now.");
     }
@@ -675,11 +701,68 @@ public class ParquetSplitReaderUtil {
           }
         }
         return new HeapRowColumnVector(batchSize, columnVectors);
+      case VARIANT:
+        validateVariantType(physicalType);
+        return new HeapRowColumnVector(
+            batchSize,
+            new HeapBytesVector(batchSize),
+            new HeapBytesVector(batchSize));
       default:
         throw new UnsupportedOperationException(fieldType + " is not supported now.");
     }
   }
 
+  private static ColumnDescriptor getVariantColumnDescriptor(
+      Type physicalType,
+      List<ColumnDescriptor> descriptors,
+      int depth,
+      String fieldName) {
+    return descriptors.stream()
+        .filter(descriptor -> descriptor.getPath().length > depth + 1
+            && fieldName.equals(descriptor.getPath()[depth + 1]))
+        .findFirst()
+        .orElseThrow(() -> new IllegalArgumentException(
+            "Invalid Variant Parquet schema: missing binary field '" + fieldName + "'."));
+  }
+
+  private static void validateVariantType(Type physicalType) {
+    if (!physicalType.isPrimitive()) {
+      GroupType groupType = physicalType.asGroupType();
+      if (isShreddedVariant(groupType)) {
+        throw new UnsupportedOperationException(
+            "Shredded Variant is not supported in Flink. "
+                + "The Parquet group '" + groupType.getName() + "' contains a '"
+                + HoodieSchema.Variant.VARIANT_TYPED_VALUE_FIELD
+                + "' field indicating a shredded layout.");
+      }
+      validateVariantField(groupType, HoodieSchema.Variant.VARIANT_VALUE_FIELD);
+      validateVariantField(groupType, HoodieSchema.Variant.VARIANT_METADATA_FIELD);
+    } else {
+      throw new IllegalArgumentException(
+          "Type mismatch, expected Variant but got '" + physicalType + "'.");
+    }
+  }
+
+  /**
+   * Checks whether a variant group contains a {@code typed_value} field, indicating a shredded
+   * layout.
+   */
+  private static boolean isShreddedVariant(GroupType groupType) {
+    return groupType.containsField(HoodieSchema.Variant.VARIANT_TYPED_VALUE_FIELD);
+  }
+
+  private static void validateVariantField(GroupType groupType, String fieldName) {
+    if (groupType.containsField(fieldName)) {
+      Type fieldType = groupType.getType(fieldName);
+      if (fieldType.isPrimitive()
+          && fieldType.asPrimitiveType().getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.BINARY) {
+        return;
+      }
+    }
+    throw new IllegalArgumentException(
+        "Invalid Variant Parquet schema: missing binary field '" + fieldName + "'.");
+  }
+
   /**
    * Returns the field index with given physical row type {@code groupType} and field name {@code fieldName}.
    *
diff --git a/hudi-flink-datasource/hudi-flink2.1.x/src/test/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializerSnapshot.java b/hudi-flink-datasource/hudi-flink2.1.x/src/test/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializerSnapshot.java
index 1a256ffe7bba..aac9a4651aff 100644
--- a/hudi-flink-datasource/hudi-flink2.1.x/src/test/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializerSnapshot.java
+++ b/hudi-flink-datasource/hudi-flink2.1.x/src/test/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializerSnapshot.java
@@ -34,6 +34,9 @@ public class KryoSerializerSnapshot<T> implements TypeSerializerSnapshot<T> {
 
   private Class<T> type;
 
+  public KryoSerializerSnapshot() {
+  }
+
   public KryoSerializerSnapshot(Class<T> type) {
     this.type = type;
   }
diff --git a/hudi-flink-datasource/hudi-flink2.1.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java b/hudi-flink-datasource/hudi-flink2.1.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
new file mode 100644
index 000000000000..2727b40b1393
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink2.1.x/src/test/java/org/apache/hudi/adapter/DataTypeAdapterTestUtils.java
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+import org.apache.flink.types.variant.BinaryVariant;
+
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+
+/**
+ * Adapter utils.
+ */
+public class DataTypeAdapterTestUtils {
+  public static void assertAsBinaryVariant(Object variantObject) {
+    assertInstanceOf(BinaryVariant.class, variantObject, "Variant column should be a BinaryVariant");
+  }
+}
diff --git a/pom.xml b/pom.xml
index cd26cc509f36..0e556cc76c1c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -153,14 +153,14 @@
     <flink1.19.version>1.19.2</flink1.19.version>
     <flink1.18.version>1.18.1</flink1.18.version>
     <flink1.17.version>1.17.1</flink1.17.version>
-    <flink.version>${flink1.20.version}</flink.version>
-    <hudi.flink.module>hudi-flink1.20.x</hudi.flink.module>
-    <flink.bundle.version>1.20</flink.bundle.version>
+    <flink.version>${flink2.1.version}</flink.version>
+    <hudi.flink.module>hudi-flink2.1.x</hudi.flink.module>
+    <flink.bundle.version>2.1</flink.bundle.version>
     <!-- This is fixed to match with version from flink-avro -->
     <flink.avro.version>1.11.4</flink.avro.version>
-    <flink.format.parquet.version>1.13.1</flink.format.parquet.version>
+    <flink.format.parquet.version>1.15.2</flink.format.parquet.version>
     <!-- check kafka version -->
-    <flink.connector.kafka.version>3.3.0-1.20</flink.connector.kafka.version>
+    <flink.connector.kafka.version>4.0.1-2.0</flink.connector.kafka.version>
     <flink.runtime.artifactId>flink-runtime</flink.runtime.artifactId>
     <flink.table.runtime.artifactId>flink-table-runtime</flink.table.runtime.artifactId>
     <flink.table.planner.artifactId>flink-table-planner_2.12</flink.table.planner.artifactId>
@@ -170,7 +170,7 @@
     <flink.streaming.java.artifactId>flink-streaming-java</flink.streaming.java.artifactId>
     <flink.clients.artifactId>flink-clients</flink.clients.artifactId>
     <flink.connector.kafka.artifactId>flink-connector-kafka</flink.connector.kafka.artifactId>
-    <flink.hadoop.compatibility.artifactId>flink-hadoop-compatibility_2.12</flink.hadoop.compatibility.artifactId>
+    <flink.hadoop.compatibility.artifactId>flink-hadoop-compatibility</flink.hadoop.compatibility.artifactId>
     <rocksdbjni.version>7.5.3</rocksdbjni.version>
     <spark33.version>3.3.4</spark33.version>
     <spark34.version>3.4.3</spark34.version>
@@ -2899,6 +2899,7 @@
         <module>hudi-flink-datasource/hudi-flink2.1.x</module>
       </modules>
       <activation>
+        <activeByDefault>true</activeByDefault>
         <property>
           <name>flink2.1</name>
         </property>
@@ -2942,7 +2943,6 @@
         <module>hudi-flink-datasource/hudi-flink1.20.x</module>
       </modules>
       <activation>
-        <activeByDefault>true</activeByDefault>
         <property>
           <name>flink1.20</name>
         </property>


(hudi) 04/16: refactor: move checkpoint metadata lookup helper to hudi-common (#18489)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 481eb8047c65d431c1994a780c4196e7ad888a45
Author: Surya Prasanna <sy...@uber.com>
AuthorDate: Mon May 4 13:42:43 2026 -0700

    refactor: move checkpoint metadata lookup helper to hudi-common (#18489)
    
    This PR moves the checkpoint metadata lookup helper into hudi-common so ingestion-related code can reuse the same timeline utility instead of keeping the logic in utilities-only code.
    
    (cherry picked from commit 471bb48338bd9642bd593472b029915180156038)
---
 .../hudi/common/table/timeline/TimelineUtils.java  | 27 ++++++++++++++++++++++
 .../streamer/StreamerCheckpointUtils.java          | 22 ++++++------------
 2 files changed, 34 insertions(+), 15 deletions(-)

diff --git a/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java b/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
index cfc1520b2393..4b2e6ee55fc6 100644
--- a/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
+++ b/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
@@ -49,6 +49,7 @@ import java.io.IOException;
 import java.io.InputStream;
 import java.text.ParseException;
 import java.util.AbstractMap;
+import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.Date;
@@ -647,4 +648,30 @@ public class TimelineUtils {
     }
     return writerOption;
   }
+
+  /**
+   * Returns the latest reverse-ordered instant whose commit metadata contains at least one of the provided
+   * checkpoint metadata keys.
+   *
+   * @param timeline timeline to scan; expected to contain completed instants
+   * @param checkpointKeys checkpoint metadata keys to look for in commit metadata
+   * @return an {@link Option} containing a {@link Pair} of the matching instant's
+   *         {@link HoodieInstant#toString()} value and its
+   *         {@link HoodieCommitMetadata}; {@link Option#empty()} if no matching instant is found
+   * @throws IOException if reading commit metadata fails
+   */
+  @SuppressWarnings("unchecked")
+  public static Option<Pair<String, HoodieCommitMetadata>> getLatestInstantAndCommitMetadataWithValidCheckpointInfo(
+      HoodieTimeline timeline, String... checkpointKeys) throws IOException {
+    return (Option<Pair<String, HoodieCommitMetadata>>) timeline.getReverseOrderedInstants().map(instant -> {
+      try {
+        HoodieCommitMetadata commitMetadata = timeline.readCommitMetadata(instant);
+        boolean hasCheckpointMetadata = Arrays.stream(checkpointKeys)
+            .anyMatch(key -> !StringUtils.isNullOrEmpty(commitMetadata.getMetadata(key)));
+        return hasCheckpointMetadata ? Option.of(Pair.of(instant.toString(), commitMetadata)) : Option.empty();
+      } catch (IOException e) {
+        throw new HoodieIOException("Failed to parse HoodieCommitMetadata for " + instant.toString(), e);
+      }
+    }).filter(Option::isPresent).findFirst().orElse(Option.empty());
+  }
 }
diff --git a/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamerCheckpointUtils.java b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamerCheckpointUtils.java
index 03db67b9d03f..560d9fd2172a 100644
--- a/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamerCheckpointUtils.java
+++ b/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamerCheckpointUtils.java
@@ -28,6 +28,7 @@ import org.apache.hudi.common.table.checkpoint.Checkpoint;
 import org.apache.hudi.common.table.checkpoint.CheckpointUtils;
 import org.apache.hudi.common.table.timeline.HoodieInstant;
 import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineUtils;
 import org.apache.hudi.common.util.ConfigUtils;
 import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.StringUtils;
@@ -201,21 +202,12 @@ public class StreamerCheckpointUtils {
 
   public static Option<Pair<String, HoodieCommitMetadata>> getLatestInstantAndCommitMetadataWithValidCheckpointInfo(HoodieTimeline timeline)
       throws IOException {
-    return (Option<Pair<String, HoodieCommitMetadata>>) timeline.getReverseOrderedInstants().map(instant -> {
-      try {
-        HoodieCommitMetadata commitMetadata = timeline.readCommitMetadata(instant);
-        if (!StringUtils.isNullOrEmpty(commitMetadata.getMetadata(HoodieStreamer.CHECKPOINT_KEY))
-            || !StringUtils.isNullOrEmpty(commitMetadata.getMetadata(HoodieStreamer.CHECKPOINT_RESET_KEY))
-            || !StringUtils.isNullOrEmpty(commitMetadata.getMetadata(STREAMER_CHECKPOINT_KEY_V2))
-            || !StringUtils.isNullOrEmpty(commitMetadata.getMetadata(STREAMER_CHECKPOINT_RESET_KEY_V2))) {
-          return Option.of(Pair.of(instant.toString(), commitMetadata));
-        } else {
-          return Option.empty();
-        }
-      } catch (IOException e) {
-        throw new HoodieIOException("Failed to parse HoodieCommitMetadata for " + instant.toString(), e);
-      }
-    }).filter(Option::isPresent).findFirst().orElse(Option.empty());
+    return TimelineUtils.getLatestInstantAndCommitMetadataWithValidCheckpointInfo(
+        timeline,
+        HoodieStreamer.CHECKPOINT_KEY,
+        HoodieStreamer.CHECKPOINT_RESET_KEY,
+        STREAMER_CHECKPOINT_KEY_V2,
+        STREAMER_CHECKPOINT_RESET_KEY_V2);
   }
 
   public static Option<HoodieCommitMetadata> getLatestCommitMetadataWithValidCheckpointInfo(HoodieTimeline timeline) throws IOException {


(hudi) 06/16: feat(flink): Wire Flink 2.1 nested Parquet readers into the Hudi read path (FLINK-35702) (#18700)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 08584dabc8c4c4036b9f6d7eaf2f81a43578cfe9
Author: Shihuan Liu <sk...@gmail.com>
AuthorDate: Thu May 7 20:07:05 2026 -0700

    feat(flink): Wire Flink 2.1 nested Parquet readers into the Hudi read path (FLINK-35702) (#18700)
    
    (cherry picked from commit 47bf4e41342e4f1dab26a7fb1489f278fbd1226c)
---
 .../table/format/cow/ParquetSplitReaderUtil.java   | 686 ++++++++++++---------
 .../format/cow/vector/HeapMapColumnVector.java     |  19 +-
 .../cow/vector/reader/NestedColumnReader.java      |  22 +
 .../reader/ParquetColumnarRowSplitReader.java      |  34 +-
 4 files changed, 472 insertions(+), 289 deletions(-)

diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/ParquetSplitReaderUtil.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/ParquetSplitReaderUtil.java
index 2bb5be1d9614..5468dc86a25a 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/ParquetSplitReaderUtil.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/ParquetSplitReaderUtil.java
@@ -7,7 +7,7 @@
  * "License"); you may not use this file except in compliance
  * with the License.  You may obtain a copy of the License at
  *
- *     http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
@@ -19,19 +19,18 @@
 package org.apache.hudi.table.format.cow;
 
 import org.apache.hudi.common.util.ValidationUtils;
-import org.apache.hudi.table.format.cow.vector.HeapArrayGroupColumnVector;
 import org.apache.hudi.table.format.cow.vector.HeapArrayVector;
 import org.apache.hudi.table.format.cow.vector.HeapDecimalVector;
 import org.apache.hudi.table.format.cow.vector.HeapMapColumnVector;
 import org.apache.hudi.table.format.cow.vector.HeapRowColumnVector;
-import org.apache.hudi.table.format.cow.vector.reader.ArrayColumnReader;
-import org.apache.hudi.table.format.cow.vector.reader.ArrayGroupReader;
 import org.apache.hudi.table.format.cow.vector.reader.EmptyColumnReader;
 import org.apache.hudi.table.format.cow.vector.reader.FixedLenBytesColumnReader;
 import org.apache.hudi.table.format.cow.vector.reader.Int64TimestampColumnReader;
-import org.apache.hudi.table.format.cow.vector.reader.MapColumnReader;
+import org.apache.hudi.table.format.cow.vector.reader.NestedColumnReader;
 import org.apache.hudi.table.format.cow.vector.reader.ParquetColumnarRowSplitReader;
-import org.apache.hudi.table.format.cow.vector.reader.RowColumnReader;
+import org.apache.hudi.table.format.cow.vector.type.ParquetField;
+import org.apache.hudi.table.format.cow.vector.type.ParquetGroupField;
+import org.apache.hudi.table.format.cow.vector.type.ParquetPrimitiveField;
 
 import org.apache.flink.core.fs.Path;
 import org.apache.flink.formats.parquet.vector.reader.BooleanColumnReader;
@@ -64,12 +63,14 @@ import org.apache.flink.table.types.logical.DecimalType;
 import org.apache.flink.table.types.logical.IntType;
 import org.apache.flink.table.types.logical.LocalZonedTimestampType;
 import org.apache.flink.table.types.logical.LogicalType;
-import org.apache.flink.table.types.logical.LogicalTypeFamily;
-import org.apache.flink.table.types.logical.LogicalTypeRoot;
 import org.apache.flink.table.types.logical.MapType;
+import org.apache.flink.table.types.logical.MultisetType;
 import org.apache.flink.table.types.logical.RowType;
 import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.util.FlinkRuntimeException;
 import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.StringUtils;
+
 import org.apache.hadoop.conf.Configuration;
 import org.apache.parquet.ParquetRuntimeException;
 import org.apache.parquet.column.ColumnDescriptor;
@@ -77,12 +78,18 @@ import org.apache.parquet.column.page.PageReadStore;
 import org.apache.parquet.column.page.PageReader;
 import org.apache.parquet.filter.UnboundRecordFilter;
 import org.apache.parquet.filter2.predicate.FilterPredicate;
+import org.apache.parquet.io.ColumnIO;
+import org.apache.parquet.io.GroupColumnIO;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.PrimitiveColumnIO;
 import org.apache.parquet.schema.GroupType;
 import org.apache.parquet.schema.InvalidSchemaException;
 import org.apache.parquet.schema.OriginalType;
 import org.apache.parquet.schema.PrimitiveType;
 import org.apache.parquet.schema.Type;
 
+import javax.annotation.Nullable;
+
 import java.io.IOException;
 import java.math.BigDecimal;
 import java.sql.Date;
@@ -90,25 +97,38 @@ import java.time.LocalDate;
 import java.time.LocalDateTime;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 import java.util.Map;
+import java.util.Objects;
 import java.util.stream.Collectors;
 
 import static org.apache.flink.table.utils.DateTimeUtils.toInternal;
 import static org.apache.hudi.common.util.StringUtils.getUTF8Bytes;
 import static org.apache.parquet.Preconditions.checkArgument;
+import static org.apache.parquet.schema.Type.Repetition.REPEATED;
+import static org.apache.parquet.schema.Type.Repetition.REQUIRED;
 
 /**
  * Util for generating {@link ParquetColumnarRowSplitReader}.
  *
- * <p>NOTE: reference from Flink release 1.11.2 {@code ParquetSplitReaderUtil}, modify to support INT64
- * based TIMESTAMP_MILLIS as ConvertedType, should remove when Flink supports that.
+ * <p>Uses the Dremel-style nested reader ported from Apache Flink 2.1 (FLINK-35702). For primitive
+ * top-level columns we keep Hudi's specialized readers β€” {@link Int64TimestampColumnReader},
+ * {@link FixedLenBytesColumnReader}, and the Hudi {@link HeapDecimalVector} β€” unchanged. For
+ * nested types (ARRAY / MAP / MULTISET / ROW) we build a {@link ParquetField} tree once per
+ * split via {@link #buildFieldsList(List, List, MessageColumnIO)} and delegate reading to
+ * {@link NestedColumnReader}.
+ *
+ * <p>Schema evolution: missing top-level fields are still handled by the caller
+ * ({@link ParquetColumnarRowSplitReader} patches them with null vectors). Missing fields
+ * <em>inside</em> a Row are handled here β€” {@link #constructField} returns {@code null} for a
+ * child that isn't physically present, and the corresponding child in the pre-allocated vector
+ * is filled with nulls via {@link #createVectorFromConstant} so the Dremel assembler can
+ * passthrough the slot (see {@link NestedColumnReader#readToVector}).
  */
 public class ParquetSplitReaderUtil {
 
-  /**
-   * Util for generating partitioned {@link ParquetColumnarRowSplitReader}.
-   */
+  /** Util for generating partitioned {@link ParquetColumnarRowSplitReader}. */
   public static ParquetColumnarRowSplitReader genPartColumnarRowReader(
       boolean utcTimestamp,
       boolean caseSensitive,
@@ -182,10 +202,13 @@ public class ParquetSplitReaderUtil {
     return readVector;
   }
 
-  private static ColumnVector createVectorFromConstant(
-      LogicalType type,
-      Object value,
-      int batchSize) {
+  /**
+   * Builds a constant-filled column vector for either a partition column (non-null value) or a
+   * missing-column slot (null value). Used both at the batch-generator level for partition
+   * injection and at the row-reader level for fields absent from the Parquet file.
+   */
+  public static ColumnVector createVectorFromConstant(
+      LogicalType type, Object value, int batchSize) {
     switch (type.getTypeRoot()) {
       case CHAR:
       case VARCHAR:
@@ -278,6 +301,7 @@ public class ParquetSplitReaderUtil {
             value == null ? null : toInternal((Date) value),
             batchSize);
       case TIMESTAMP_WITHOUT_TIME_ZONE:
+      case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
         HeapTimestampVector tv = new HeapTimestampVector(batchSize);
         if (value == null) {
           tv.fillWithNulls();
@@ -286,46 +310,41 @@ public class ParquetSplitReaderUtil {
         }
         return tv;
       case ARRAY:
-        ArrayType arrayType = (ArrayType) type;
-        if (arrayType.getElementType().isAnyOf(LogicalTypeFamily.CONSTRUCTED)) {
-          HeapArrayGroupColumnVector arrayGroup = new HeapArrayGroupColumnVector(batchSize);
-          if (value == null) {
-            arrayGroup.fillWithNulls();
-            return arrayGroup;
-          } else {
-            throw new UnsupportedOperationException("Unsupported create array with default value.");
-          }
-        } else {
-          HeapArrayVector arrayVector = new HeapArrayVector(batchSize);
-          if (value == null) {
-            arrayVector.fillWithNulls();
-            return arrayVector;
-          } else {
-            throw new UnsupportedOperationException("Unsupported create array with default value.");
-          }
+        if (value != null) {
+          throw new UnsupportedOperationException("Unsupported create array with default value.");
         }
+        HeapArrayVector arrayVector = new HeapArrayVector(batchSize);
+        arrayVector.fillWithNulls();
+        return arrayVector;
       case MAP:
-        HeapMapColumnVector mapVector = new HeapMapColumnVector(batchSize, null, null);
-        if (value == null) {
-          mapVector.fillWithNulls();
-          return mapVector;
-        } else {
-          throw new UnsupportedOperationException("Unsupported create map with default value.");
+      case MULTISET:
+        if (value != null) {
+          throw new UnsupportedOperationException(
+              "Unsupported create " + type.getTypeRoot() + " with default value.");
         }
+        HeapMapColumnVector mapVector = new HeapMapColumnVector(batchSize, null, null);
+        mapVector.fillWithNulls();
+        return mapVector;
       case ROW:
-        HeapRowColumnVector rowVector = new HeapRowColumnVector(batchSize);
-        if (value == null) {
-          rowVector.fillWithNulls();
-          return rowVector;
-        } else {
+        if (value != null) {
           throw new UnsupportedOperationException("Unsupported create row with default value.");
         }
+        RowType rowType = (RowType) type;
+        WritableColumnVector[] childVectors = new WritableColumnVector[rowType.getFieldCount()];
+        for (int i = 0; i < childVectors.length; i++) {
+          childVectors[i] =
+              (WritableColumnVector) createVectorFromConstant(rowType.getTypeAt(i), null, batchSize);
+        }
+        HeapRowColumnVector rowVector = new HeapRowColumnVector(batchSize, childVectors);
+        rowVector.fillWithNulls();
+        return rowVector;
       default:
         throw new UnsupportedOperationException("Unsupported type: " + type);
     }
   }
 
-  private static List<ColumnDescriptor> filterDescriptors(int depth, Type type, List<ColumnDescriptor> columns) throws ParquetRuntimeException {
+  private static List<ColumnDescriptor> filterDescriptors(
+      int depth, Type type, List<ColumnDescriptor> columns) throws ParquetRuntimeException {
     List<ColumnDescriptor> filtered = new ArrayList<>();
     for (ColumnDescriptor descriptor : columns) {
       if (depth >= descriptor.getPath().length) {
@@ -339,24 +358,61 @@ public class ParquetSplitReaderUtil {
     return filtered;
   }
 
+  /**
+   * Creates a {@link ColumnReader} for one top-level requested field. For primitive types the
+   * Hudi-specialized reader path is used. For nested types ({@code ARRAY}, {@code MAP},
+   * {@code MULTISET}, {@code ROW}) the Dremel-style {@link NestedColumnReader} is used, driven by
+   * the supplied pre-built {@link ParquetField} tree.
+   *
+   * @param field the {@link ParquetField} tree for this column, built by
+   *     {@link #buildFieldsList(List, List, MessageColumnIO)}. Required (non-null) for nested
+   *     types; ignored for primitives.
+   */
+  public static ColumnReader createColumnReader(
+      boolean utcTimestamp,
+      LogicalType fieldType,
+      Type physicalType,
+      List<ColumnDescriptor> descriptors,
+      PageReadStore pages,
+      @Nullable ParquetField field) throws IOException {
+    switch (fieldType.getTypeRoot()) {
+      case ARRAY:
+      case MAP:
+      case MULTISET:
+      case ROW:
+        Preconditions.checkNotNull(
+            field, "ParquetField must be provided for nested type: %s", fieldType);
+        return new NestedColumnReader(utcTimestamp, pages, field);
+      default:
+        return createPrimitiveColumnReader(utcTimestamp, fieldType, physicalType, descriptors, pages);
+    }
+  }
+
+  /**
+   * Backward-compat entry point kept for callers that don't project nested types and therefore
+   * never need a {@link ParquetField} tree. Forwards to the {@link ParquetField}-aware overload
+   * with a null field; nested types now go through that overload directly.
+   *
+   * @deprecated use {@link #createColumnReader(boolean, LogicalType, Type, List, PageReadStore,
+   *     ParquetField)} so nested types take the Dremel path.
+   */
+  @Deprecated
   public static ColumnReader createColumnReader(
       boolean utcTimestamp,
       LogicalType fieldType,
       Type physicalType,
       List<ColumnDescriptor> descriptors,
       PageReadStore pages) throws IOException {
-    return createColumnReader(utcTimestamp, fieldType, physicalType, descriptors,
-        pages, 0);
+    return createColumnReader(utcTimestamp, fieldType, physicalType, descriptors, pages, null);
   }
 
-  private static ColumnReader createColumnReader(
+  private static ColumnReader createPrimitiveColumnReader(
       boolean utcTimestamp,
       LogicalType fieldType,
       Type physicalType,
       List<ColumnDescriptor> columns,
-      PageReadStore pages,
-      int depth) throws IOException {
-    List<ColumnDescriptor> descriptors = filterDescriptors(depth, physicalType, columns);
+      PageReadStore pages) throws IOException {
+    List<ColumnDescriptor> descriptors = filterDescriptors(0, physicalType, columns);
     ColumnDescriptor descriptor = descriptors.get(0);
     PageReader pageReader = pages.getPageReader(descriptor);
     switch (fieldType.getTypeRoot()) {
@@ -392,7 +448,9 @@ public class ParquetSplitReaderUtil {
           case INT96:
             return new TimestampColumnReader(utcTimestamp, descriptor, pageReader);
           default:
-            throw new AssertionError();
+            throw new AssertionError(
+                "Unexpected physical type for TIMESTAMP: "
+                    + descriptor.getPrimitiveType().getPrimitiveTypeName());
         }
       case DECIMAL:
         switch (descriptor.getPrimitiveType().getPrimitiveTypeName()) {
@@ -403,106 +461,23 @@ public class ParquetSplitReaderUtil {
           case BINARY:
             return new BytesColumnReader(descriptor, pageReader);
           case FIXED_LEN_BYTE_ARRAY:
-            return new FixedLenBytesColumnReader(
-                descriptor, pageReader);
+            return new FixedLenBytesColumnReader(descriptor, pageReader);
           default:
-            throw new AssertionError();
-        }
-      case ARRAY:
-        ArrayType arrayType = (ArrayType) fieldType;
-        if (arrayType.getElementType().isAnyOf(LogicalTypeFamily.CONSTRUCTED)) {
-          boolean isThreeLevelList = isThreeLevelList(physicalType);
-          // 3-level List structure, drill down 2 level to get type for `element`
-          Type elementType = isThreeLevelList
-              ? physicalType.asGroupType().getType(0).asGroupType().getType(0)
-              : physicalType.asGroupType().getType(0);
-          int elementDepth = isThreeLevelList ? depth + 2 : depth + 1;
-          return new ArrayGroupReader(createColumnReader(
-              utcTimestamp,
-              arrayType.getElementType(),
-              elementType,
-              descriptors,
-              pages,
-              elementDepth));
-        } else {
-          return new ArrayColumnReader(
-              descriptor,
-              pageReader,
-              utcTimestamp,
-              descriptor.getPrimitiveType(),
-              fieldType);
-        }
-      case MAP:
-        MapType mapType = (MapType) fieldType;
-        ArrayColumnReader keyReader =
-            new ArrayColumnReader(
-                descriptor,
-                pageReader,
-                utcTimestamp,
-                descriptor.getPrimitiveType(),
-                new ArrayType(mapType.getKeyType()));
-        ColumnReader<WritableColumnVector> valueReader;
-        if (mapType.getValueType().isAnyOf(LogicalTypeFamily.CONSTRUCTED)) {
-          valueReader = new ArrayGroupReader(createColumnReader(
-              utcTimestamp,
-              mapType.getValueType(),
-              physicalType.asGroupType().getType(0).asGroupType().getType(1), // Get the value physical type
-              descriptors.subList(1, descriptors.size()), // remove the key descriptor
-              pages,
-              depth + 2)); // increase the depth by 2, because there's a key_value entry in the path
-        } else {
-          valueReader = new ArrayColumnReader(
-              descriptors.get(1),
-              pages.getPageReader(descriptors.get(1)),
-              utcTimestamp,
-              descriptors.get(1).getPrimitiveType(),
-              new ArrayType(mapType.getValueType()));
+            throw new AssertionError(
+                "Unexpected physical type for DECIMAL: "
+                    + descriptor.getPrimitiveType().getPrimitiveTypeName());
         }
-        return new MapColumnReader(keyReader, valueReader);
-      case ROW:
-        RowType rowType = (RowType) fieldType;
-        GroupType groupType = physicalType.asGroupType();
-        List<ColumnReader> fieldReaders = new ArrayList<>();
-        for (int i = 0; i < rowType.getFieldCount(); i++) {
-          // schema evolution: read the parquet file with a new extended field name.
-          int fieldIndex = getFieldIndexInPhysicalType(rowType.getFields().get(i).getName(), groupType);
-          if (fieldIndex < 0) {
-            fieldReaders.add(new EmptyColumnReader());
-          } else {
-            // Check for nested row in array with atomic field type.
-
-            // This is done to meet the Parquet field algorithm that pushes multiplicity and structures down to individual fields.
-            // In Parquet, an array of rows is stored as separate arrays for each field.
-
-            // Limitations: It won't work for multiple nested arrays and maps.
-            // The main problem is that the Flink classes and interface don't follow that pattern.
-            if (descriptors.get(fieldIndex).getMaxRepetitionLevel() > 0 && !rowType.getTypeAt(i).is(LogicalTypeRoot.ARRAY)) {
-              fieldReaders.add(
-                  createColumnReader(
-                      utcTimestamp,
-                      new ArrayType(rowType.getTypeAt(i).isNullable(), rowType.getTypeAt(i)),
-                      groupType.getType(fieldIndex),
-                      descriptors,
-                      pages,
-                      depth + 1));
-            } else {
-              fieldReaders.add(
-                  createColumnReader(
-                      utcTimestamp,
-                      rowType.getTypeAt(i),
-                      groupType.getType(fieldIndex),
-                      descriptors,
-                      pages,
-                      depth + 1));
-            }
-          }
-        }
-        return new RowColumnReader(fieldReaders);
       default:
         throw new UnsupportedOperationException(fieldType + " is not supported now.");
     }
   }
 
+  /**
+   * Creates the writable column vector that the reader will write into. The returned vector shape
+   * matches {@code fieldType}; for ROW types missing physical fields are slotted with null-filled
+   * vectors (sourced from {@link #createVectorFromConstant}) so that the Dremel assembler in
+   * {@link NestedColumnReader} can pass them through unchanged.
+   */
   public static WritableColumnVector createWritableColumnVector(
       int batchSize,
       LogicalType fieldType,
@@ -523,40 +498,48 @@ public class ParquetSplitReaderUtil {
     switch (fieldType.getTypeRoot()) {
       case BOOLEAN:
         checkArgument(
-            typeName == PrimitiveType.PrimitiveTypeName.BOOLEAN, getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
+            typeName == PrimitiveType.PrimitiveTypeName.BOOLEAN,
+            getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
         return new HeapBooleanVector(batchSize);
       case TINYINT:
         checkArgument(
-            typeName == PrimitiveType.PrimitiveTypeName.INT32, getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
+            typeName == PrimitiveType.PrimitiveTypeName.INT32,
+            getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
         return new HeapByteVector(batchSize);
       case DOUBLE:
         checkArgument(
-            typeName == PrimitiveType.PrimitiveTypeName.DOUBLE, getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
+            typeName == PrimitiveType.PrimitiveTypeName.DOUBLE,
+            getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
         return new HeapDoubleVector(batchSize);
       case FLOAT:
         checkArgument(
-            typeName == PrimitiveType.PrimitiveTypeName.FLOAT, getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
+            typeName == PrimitiveType.PrimitiveTypeName.FLOAT,
+            getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
         return new HeapFloatVector(batchSize);
       case INTEGER:
       case DATE:
       case TIME_WITHOUT_TIME_ZONE:
         checkArgument(
-            typeName == PrimitiveType.PrimitiveTypeName.INT32, getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
+            typeName == PrimitiveType.PrimitiveTypeName.INT32,
+            getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
         return new HeapIntVector(batchSize);
       case BIGINT:
         checkArgument(
-            typeName == PrimitiveType.PrimitiveTypeName.INT64, getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
+            typeName == PrimitiveType.PrimitiveTypeName.INT64,
+            getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
         return new HeapLongVector(batchSize);
       case SMALLINT:
         checkArgument(
-            typeName == PrimitiveType.PrimitiveTypeName.INT32, getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
+            typeName == PrimitiveType.PrimitiveTypeName.INT32,
+            getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
         return new HeapShortVector(batchSize);
       case CHAR:
       case VARCHAR:
       case BINARY:
       case VARBINARY:
         checkArgument(
-            typeName == PrimitiveType.PrimitiveTypeName.BINARY, getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
+            typeName == PrimitiveType.PrimitiveTypeName.BINARY,
+            getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
         return new HeapBytesVector(batchSize);
       case TIMESTAMP_WITHOUT_TIME_ZONE:
       case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
@@ -566,112 +549,64 @@ public class ParquetSplitReaderUtil {
       case DECIMAL:
         checkArgument(
             (typeName == PrimitiveType.PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY
-                || typeName == PrimitiveType.PrimitiveTypeName.BINARY)
+                    || typeName == PrimitiveType.PrimitiveTypeName.BINARY)
                 && primitiveType.getOriginalType() == OriginalType.DECIMAL,
             getPrimitiveTypeCheckFailureMessage(typeName, fieldType));
         return new HeapDecimalVector(batchSize);
       case ARRAY:
         ArrayType arrayType = (ArrayType) fieldType;
-        if (arrayType.getElementType().isAnyOf(LogicalTypeFamily.CONSTRUCTED)) {
-          boolean isThreeLevelList = isThreeLevelList(physicalType);
-          // 3-level List structure, drill down 2 level to get type for `element`
-          Type elementType = isThreeLevelList
-              ? physicalType.asGroupType().getType(0).asGroupType().getType(0)
-              : physicalType.asGroupType().getType(0);
-          int elementDepth = isThreeLevelList ? depth + 2 : depth + 1;
-          return new HeapArrayGroupColumnVector(
-              batchSize,
-              createWritableColumnVector(
-                  batchSize,
-                  arrayType.getElementType(),
-                  elementType,
-                  descriptors,
-                  elementDepth));
-        } else {
-          return new HeapArrayVector(
-              batchSize,
-              createWritableColumnVector(
-                  batchSize,
-                  arrayType.getElementType(),
-                  physicalType,
-                  descriptors,
-                  depth));
-        }
-      case MAP:
+        return new HeapArrayVector(
+            batchSize,
+            createWritableColumnVector(
+                batchSize, arrayType.getElementType(), physicalType, descriptors, depth));
+      case MAP: {
         MapType mapType = (MapType) fieldType;
-        GroupType repeatedType = physicalType.asGroupType().getType(0).asGroupType();
-        // the map column has three level paths.
-        WritableColumnVector keyColumnVector = createWritableColumnVector(
+        GroupType repeatedType = unwrapMapRepeatedType(physicalType);
+        return new HeapMapColumnVector(
             batchSize,
-            new ArrayType(mapType.getKeyType().isNullable(), mapType.getKeyType()),
-            repeatedType.getType(0),
-            descriptors,
-            depth + 2);
-        WritableColumnVector valueColumnVector;
-        if (mapType.getValueType().isAnyOf(LogicalTypeFamily.CONSTRUCTED)) {
-          valueColumnVector = new HeapArrayGroupColumnVector(
-              batchSize,
-              createWritableColumnVector(
-                  batchSize,
-                  mapType.getValueType(),
-                  repeatedType.getType(1).asGroupType(),
-                  descriptors,
-                  depth + 2));
-        } else {
-          valueColumnVector = createWritableColumnVector(
-              batchSize,
-              new ArrayType(mapType.getValueType().isNullable(), mapType.getValueType()),
-              repeatedType.getType(1),
-              descriptors,
-              depth + 2);
-        }
-        return new HeapMapColumnVector(batchSize, keyColumnVector, valueColumnVector);
+            createWritableColumnVector(
+                batchSize, mapType.getKeyType(), repeatedType.getType(0), descriptors, depth + 2),
+            createWritableColumnVector(
+                batchSize, mapType.getValueType(), repeatedType.getType(1), descriptors, depth + 2));
+      }
+      case MULTISET: {
+        MultisetType multisetType = (MultisetType) fieldType;
+        GroupType repeatedType = unwrapMapRepeatedType(physicalType);
+        return new HeapMapColumnVector(
+            batchSize,
+            createWritableColumnVector(
+                batchSize,
+                multisetType.getElementType(),
+                repeatedType.getType(0),
+                descriptors,
+                depth + 2),
+            createWritableColumnVector(
+                batchSize,
+                new IntType(false),
+                repeatedType.getType(1),
+                descriptors,
+                depth + 2));
+      }
       case ROW:
         RowType rowType = (RowType) fieldType;
         GroupType groupType = physicalType.asGroupType();
         WritableColumnVector[] columnVectors = new WritableColumnVector[rowType.getFieldCount()];
         for (int i = 0; i < columnVectors.length; i++) {
-          // schema evolution: read the file with a new extended field name.
           int fieldIndex = getFieldIndexInPhysicalType(rowType.getFields().get(i).getName(), groupType);
           if (fieldIndex < 0) {
-            // Check for nested row in array with atomic field type.
-
-            // This is done to meet the Parquet field algorithm that pushes multiplicity and structures down to individual fields.
-            // In Parquet, an array of rows is stored as separate arrays for each field.
-
-            // Limitations: It won't work for multiple nested arrays and maps.
-            // The main problem is that the Flink classes and interface don't follow that pattern.
-            if (groupType.getRepetition().equals(Type.Repetition.REPEATED) && !rowType.getTypeAt(i).is(LogicalTypeRoot.ARRAY)) {
-              columnVectors[i] = (WritableColumnVector) createVectorFromConstant(
-                  new ArrayType(rowType.getTypeAt(i).isNullable(), rowType.getTypeAt(i)), null, batchSize);
-            } else {
-              columnVectors[i] = (WritableColumnVector) createVectorFromConstant(rowType.getTypeAt(i), null, batchSize);
-            }
+            // Schema evolution: logical field is absent from the Parquet file. Slot a null-filled
+            // vector of the correct shape; NestedColumnReader.readRow will pass it through when the
+            // matching ParquetField child is null.
+            columnVectors[i] =
+                (WritableColumnVector) createVectorFromConstant(rowType.getTypeAt(i), null, batchSize);
           } else {
-            // Check for nested row in array with atomic field type.
-
-            // This is done to meet the Parquet field algorithm that pushes multiplicity and structures down to individual fields.
-            // In Parquet, an array of rows is stored as separate arrays for each field.
-
-            // Limitations: It won't work for multiple nested arrays and maps.
-            // The main problem is that the Flink classes and interface don't follow that pattern.
-            if (descriptors.get(fieldIndex).getMaxRepetitionLevel() > 0 && !rowType.getTypeAt(i).is(LogicalTypeRoot.ARRAY)) {
-              columnVectors[i] =
-                  createWritableColumnVector(
-                      batchSize,
-                      new ArrayType(rowType.getTypeAt(i).isNullable(), rowType.getTypeAt(i)),
-                      groupType.getType(fieldIndex),
-                      descriptors,
-                      depth + 1);
-            } else {
-              columnVectors[i] =
-                  createWritableColumnVector(
-                      batchSize,
-                      rowType.getTypeAt(i),
-                      groupType.getType(fieldIndex),
-                      descriptors,
-                      depth + 1);
-            }
+            columnVectors[i] =
+                createWritableColumnVector(
+                    batchSize,
+                    rowType.getTypeAt(i),
+                    groupType.getType(fieldIndex),
+                    descriptors,
+                    depth + 1);
           }
         }
         return new HeapRowColumnVector(batchSize, columnVectors);
@@ -681,56 +616,245 @@ public class ParquetSplitReaderUtil {
   }
 
   /**
-   * Returns the field index with given physical row type {@code groupType} and field name {@code fieldName}.
-   *
-   * @return The physical field index or -1 if the field does not exist
+   * Peels one {@code repeated group key_value} wrapper off a MAP / MULTISET physical type, matching
+   * Parquet's canonical 3-level map encoding.
    */
-  private static int getFieldIndexInPhysicalType(String fieldName, GroupType groupType) {
-    // get index from fileSchema type, else, return -1
-    return groupType.containsField(fieldName) ? groupType.getFieldIndex(fieldName) : -1;
+  private static GroupType unwrapMapRepeatedType(Type physicalType) {
+    return physicalType.asGroupType().getType(0).asGroupType();
   }
 
+  // ------------------------------------------------------------------------------------------
+  // ParquetField tree construction (vendored from Apache Flink 2.1 ParquetSplitReaderUtil)
+  //
+  // The only Hudi-specific divergence is in `constructField`: the ROW branch tolerates children
+  // missing from the Parquet file by emitting a null ParquetField child (upstream throws). This
+  // matches the Hudi schema-evolution contract and is the companion to the null-child branch in
+  // `NestedColumnReader#readRow` and the null-vector slot in `createWritableColumnVector#ROW`.
+  // ------------------------------------------------------------------------------------------
+
   /**
-   * Check whether the given list type is a three-level list type.
-   * <p>
-   * <list-repetition> group <name> (LIST) {
-   *   repeated group list {
-   *     <element-repetition> <element-type> element;
-   *   }
-   * }
-   *
-   * @param type list type
-   * @return true if the list type is a three-level list type
+   * Builds {@link ParquetField} trees β€” one per top-level projected logical column β€” that feed
+   * {@link NestedColumnReader}. The returned list mirrors the input {@code children} positionally;
+   * primitive top-level fields produce {@code null} entries (callers don't need a tree for those).
+   */
+  public static List<ParquetField> buildFieldsList(
+      List<RowType.RowField> children, List<String> fieldNames, MessageColumnIO columnIO) {
+    List<ParquetField> list = new ArrayList<>();
+    for (int i = 0; i < children.size(); i++) {
+      RowType.RowField child = children.get(i);
+      if (isNestedType(child.getType())) {
+        list.add(constructField(child, lookupColumnByName(columnIO, fieldNames.get(i))));
+      } else {
+        list.add(null);
+      }
+    }
+    return list;
+  }
+
+  private static boolean isNestedType(LogicalType type) {
+    return type instanceof RowType
+        || type instanceof ArrayType
+        || type instanceof MapType
+        || type instanceof MultisetType;
+  }
+
+  @Nullable
+  private static ParquetField constructField(RowType.RowField rowField, ColumnIO columnIO) {
+    boolean required = columnIO.getType().getRepetition() == REQUIRED;
+    int repetitionLevel = columnIO.getRepetitionLevel();
+    int definitionLevel = columnIO.getDefinitionLevel();
+    LogicalType type = rowField.getType();
+    String fieldName = rowField.getName();
+    if (type instanceof RowType) {
+      GroupColumnIO groupColumnIO = (GroupColumnIO) columnIO;
+      RowType rowType = (RowType) type;
+      List<RowType.RowField> childFields = rowType.getFields();
+      List<ParquetField> fieldsList = new ArrayList<>(childFields.size());
+      for (RowType.RowField childField : childFields) {
+        // Hudi schema evolution: a logical child may be absent from the Parquet file. In that
+        // case we emit a null ParquetField so that NestedColumnReader.readRow passes through the
+        // pre-filled null vector instead of recursing.
+        ColumnIO childIo = lookupColumnByNameOrNull(groupColumnIO, childField.getName());
+        if (childIo == null) {
+          fieldsList.add(null);
+        } else {
+          fieldsList.add(constructField(childField, childIo));
+        }
+      }
+      return new ParquetGroupField(
+          type,
+          repetitionLevel,
+          definitionLevel,
+          required,
+          Collections.unmodifiableList(fieldsList));
+    }
+
+    if (type instanceof MapType) {
+      GroupColumnIO groupColumnIO = (GroupColumnIO) columnIO;
+      GroupColumnIO keyValueColumnIO = getMapKeyValueColumn(groupColumnIO);
+      MapType mapType = (MapType) type;
+      ParquetField keyField =
+          constructField(
+              new RowType.RowField("", mapType.getKeyType()), keyValueColumnIO.getChild(0));
+      ParquetField valueField =
+          constructField(
+              new RowType.RowField("", mapType.getValueType()), keyValueColumnIO.getChild(1));
+      return new ParquetGroupField(
+          type,
+          repetitionLevel,
+          definitionLevel,
+          required,
+          Collections.unmodifiableList(Arrays.asList(keyField, valueField)));
+    }
+
+    if (type instanceof MultisetType) {
+      GroupColumnIO groupColumnIO = (GroupColumnIO) columnIO;
+      GroupColumnIO keyValueColumnIO = getMapKeyValueColumn(groupColumnIO);
+      MultisetType multisetType = (MultisetType) type;
+      ParquetField keyField =
+          constructField(
+              new RowType.RowField("", multisetType.getElementType()),
+              keyValueColumnIO.getChild(0));
+      ParquetField valueField =
+          constructField(
+              new RowType.RowField("", new IntType()), keyValueColumnIO.getChild(1));
+      return new ParquetGroupField(
+          type,
+          repetitionLevel,
+          definitionLevel,
+          required,
+          Collections.unmodifiableList(Arrays.asList(keyField, valueField)));
+    }
+
+    if (type instanceof ArrayType) {
+      ArrayType arrayType = (ArrayType) type;
+      ColumnIO elementTypeColumnIO;
+      if (columnIO instanceof GroupColumnIO) {
+        GroupColumnIO groupColumnIO = (GroupColumnIO) columnIO;
+        if (!StringUtils.isNullOrWhitespaceOnly(fieldName)) {
+          while (!Objects.equals(groupColumnIO.getName(), fieldName)) {
+            groupColumnIO = (GroupColumnIO) groupColumnIO.getChild(0);
+          }
+          elementTypeColumnIO = groupColumnIO;
+        } else {
+          if (arrayType.getElementType() instanceof RowType) {
+            elementTypeColumnIO = groupColumnIO;
+          } else {
+            elementTypeColumnIO = groupColumnIO.getChild(0);
+          }
+        }
+      } else if (columnIO instanceof PrimitiveColumnIO) {
+        elementTypeColumnIO = columnIO;
+      } else {
+        throw new FlinkRuntimeException(String.format("Unknown ColumnIO, %s", columnIO));
+      }
+
+      ParquetField elementField =
+          constructField(
+              new RowType.RowField("", arrayType.getElementType()),
+              getArrayElementColumn(elementTypeColumnIO));
+      if (repetitionLevel == elementField.getRepetitionLevel()) {
+        repetitionLevel = columnIO.getParent().getRepetitionLevel();
+      }
+      return new ParquetGroupField(
+          type,
+          repetitionLevel,
+          definitionLevel,
+          required,
+          Collections.singletonList(elementField));
+    }
+
+    PrimitiveColumnIO primitiveColumnIO = (PrimitiveColumnIO) columnIO;
+    return new ParquetPrimitiveField(
+        type, required, primitiveColumnIO.getColumnDescriptor(), primitiveColumnIO.getId());
+  }
+
+  /**
+   * Parquet column names are case-insensitive in Flink's lookup. Matches upstream
+   * {@code ParquetSplitReaderUtil.lookupColumnByName}; throws when absent.
    */
-  private static boolean isThreeLevelList(Type type) {
-    if (type.isPrimitive()) {
-      return false;
+  public static ColumnIO lookupColumnByName(GroupColumnIO groupColumnIO, String columnName) {
+    ColumnIO columnIO = lookupColumnByNameOrNull(groupColumnIO, columnName);
+    if (columnIO != null) {
+      return columnIO;
     }
-    GroupType groupType = type.asGroupType();
-    OriginalType originalType = groupType.getOriginalType();
-    return originalType == OriginalType.LIST
-        && groupType.getType(0).getName().equals("list");
+    throw new FlinkRuntimeException(
+        "Can not find column io for parquet reader. Column name: " + columnName);
   }
 
   /**
-   * Construct the error message when primitive type mismatches.
-   *
-   * @param primitiveType Primitive type
-   * @param fieldType     Logical field type
-   * @return The error message
+   * Case-insensitive column lookup that returns {@code null} when no match is found β€” the
+   * Hudi-specific companion to {@link #lookupColumnByName}, used by {@link #constructField} to
+   * emit null {@link ParquetField} children for fields absent from the Parquet file.
    */
-  private static String getPrimitiveTypeCheckFailureMessage(PrimitiveType.PrimitiveTypeName primitiveType, LogicalType fieldType) {
-    return String.format("Unexpected type exception. Primitive type: %s. Field type: %s.", primitiveType, fieldType.getTypeRoot().name());
+  @Nullable
+  private static ColumnIO lookupColumnByNameOrNull(
+      GroupColumnIO groupColumnIO, String columnName) {
+    ColumnIO columnIO = groupColumnIO.getChild(columnName);
+    if (columnIO != null) {
+      return columnIO;
+    }
+    for (int i = 0; i < groupColumnIO.getChildrenCount(); i++) {
+      if (groupColumnIO.getChild(i).getName().equalsIgnoreCase(columnName)) {
+        return groupColumnIO.getChild(i);
+      }
+    }
+    return null;
+  }
+
+  public static GroupColumnIO getMapKeyValueColumn(GroupColumnIO groupColumnIO) {
+    while (groupColumnIO.getChildrenCount() == 1) {
+      groupColumnIO = (GroupColumnIO) groupColumnIO.getChild(0);
+    }
+    return groupColumnIO;
+  }
+
+  public static ColumnIO getArrayElementColumn(ColumnIO columnIO) {
+    while (columnIO instanceof GroupColumnIO && !columnIO.getType().isRepetition(REPEATED)) {
+      columnIO = ((GroupColumnIO) columnIO).getChild(0);
+    }
+
+    // Three-level list: skip the synthetic `element` / `list` wrapper when present.
+    if (columnIO instanceof GroupColumnIO
+        && columnIO.getType().getLogicalTypeAnnotation() == null
+        && ((GroupColumnIO) columnIO).getChildrenCount() == 1
+        && !columnIO.getName().equals("array")
+        && !columnIO.getName().equals(columnIO.getParent().getName() + "_tuple")) {
+      return ((GroupColumnIO) columnIO).getChild(0);
+    }
+    return columnIO;
   }
 
   /**
-   * Construct the error message when original type mismatches.
+   * Returns the field index with given physical row type {@code groupType} and field name
+   * {@code fieldName}.
    *
-   * @param originalType Original type
-   * @param fieldType    Logical field type
-   * @return The error message
+   * @return the physical field index or -1 if the field does not exist
+   */
+  private static int getFieldIndexInPhysicalType(String fieldName, GroupType groupType) {
+    return groupType.containsField(fieldName) ? groupType.getFieldIndex(fieldName) : -1;
+  }
+
+  private static String getPrimitiveTypeCheckFailureMessage(
+      PrimitiveType.PrimitiveTypeName primitiveType, LogicalType fieldType) {
+    return String.format(
+        "Unexpected type exception. Primitive type: %s. Field type: %s.",
+        primitiveType, fieldType.getTypeRoot().name());
+  }
+
+  private static String getOriginalTypeCheckFailureMessage(
+      OriginalType originalType, LogicalType fieldType) {
+    return String.format(
+        "Unexpected type exception. Original type: %s. Field type: %s.",
+        originalType, fieldType.getTypeRoot().name());
+  }
+
+  /**
+   * Returns a synthetic null-column reader to fill missing top-level fields. Kept as a convenience
+   * for callers that need to mirror Hudi's original behaviour where a missing column produces an
+   * explicit null-valued reader rather than being omitted from the batch.
    */
-  private static String getOriginalTypeCheckFailureMessage(OriginalType originalType, LogicalType fieldType) {
-    return String.format("Unexpected type exception. Original type: %s. Field type: %s.", originalType, fieldType.getTypeRoot().name());
+  public static ColumnReader<WritableColumnVector> emptyColumnReader() {
+    return new EmptyColumnReader();
   }
 }
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapMapColumnVector.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapMapColumnVector.java
index a98bdebd707a..14aad22039e0 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapMapColumnVector.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapMapColumnVector.java
@@ -20,6 +20,7 @@ package org.apache.hudi.table.format.cow.vector;
 
 import lombok.Getter;
 import org.apache.flink.table.data.MapData;
+import org.apache.flink.table.data.columnar.ColumnarMapData;
 import org.apache.flink.table.data.columnar.vector.ColumnVector;
 import org.apache.flink.table.data.columnar.vector.MapColumnVector;
 import org.apache.flink.table.data.columnar.vector.heap.AbstractHeapVector;
@@ -27,6 +28,14 @@ import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector
 
 /**
  * This class represents a nullable heap map column vector.
+ *
+ * <p>Mirrors {@code org.apache.flink.table.data.columnar.vector.heap.HeapMapVector} from
+ * Flink 2.1 (FLINK-35702). One deliberate divergence from upstream is preserved for backward
+ * compatibility: the {@code keys} / {@code values} fields are typed
+ * {@link WritableColumnVector} rather than upstream's {@link ColumnVector}, so the existing
+ * Lombok-generated {@code getKeys()} / {@code getValues()} accessors keep their original
+ * signature. Callers wanting the Flink-2.1 contract (a {@code ColumnVector}) use
+ * {@link #getKeyColumnVector()} / {@link #getValueColumnVector()}.
  */
 public class HeapMapColumnVector extends AbstractHeapVector
     implements WritableColumnVector, MapColumnVector {
@@ -38,10 +47,8 @@ public class HeapMapColumnVector extends AbstractHeapVector
 
   // ---------------------------------------------------------------------------------------------
   // Flink 2.1 Dremel-style state. Populated by {@link
-  // org.apache.hudi.table.format.cow.vector.reader.NestedColumnReader} (FLINK-35702 port). The
-  // legacy {@link #getMap(int)} implementation below continues to use {@code ColumnarGroupMapData}
-  // β€” wiring it through these offsets/lengths happens in a follow-up PR that switches the read
-  // path. Left here so the new readers can compile against the additive surface.
+  // org.apache.hudi.table.format.cow.vector.reader.NestedColumnReader} (FLINK-35702 port) and
+  // consumed by {@link #getMap(int)}.
   // ---------------------------------------------------------------------------------------------
   private long[] offsets;
   private long[] lengths;
@@ -102,6 +109,8 @@ public class HeapMapColumnVector extends AbstractHeapVector
 
   @Override
   public MapData getMap(int rowId) {
-    return new ColumnarGroupMapData(keys, values, rowId);
+    long offset = offsets[rowId];
+    long length = lengths[rowId];
+    return new ColumnarMapData(keys, values, (int) offset, (int) length);
   }
 }
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java
index 27eab298b320..6ae1cc9492ec 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java
@@ -147,6 +147,28 @@ public class NestedColumnReader implements ColumnReader<WritableColumnVector> {
     if (rowPosition.getIsNull() != null) {
       setFieldNullFlag(rowPosition.getIsNull(), heapRowVector);
     }
+
+    // Hudi-specific: collapse a present row whose every child is null into a null row. The
+    // legacy RowColumnReader did this so that a SQL value like `row(null, null)` round-trips
+    // to NULL on read; preserve it here for backward compatibility. Diverges from Flink 2.1,
+    // which would surface it as Row(null, null). Mirrored by the integration test
+    // ITTestHoodieDataSource#testParquetNullChildColumnsRowTypes.
+    int rowCount = rowPosition.getPositionsCount();
+    for (int j = 0; j < rowCount; j++) {
+      if (heapRowVector.isNullAt(j)) {
+        continue;
+      }
+      boolean allChildrenNull = true;
+      for (WritableColumnVector child : finalChildrenVectors) {
+        if (!child.isNullAt(j)) {
+          allChildrenNull = false;
+          break;
+        }
+      }
+      if (allChildrenNull) {
+        heapRowVector.setNullAt(j);
+      }
+    }
     return Tuple2.of(levelDelegation, heapRowVector);
   }
 
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ParquetColumnarRowSplitReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ParquetColumnarRowSplitReader.java
index 3572b117a631..1826419db5d4 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ParquetColumnarRowSplitReader.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ParquetColumnarRowSplitReader.java
@@ -18,7 +18,9 @@
 
 package org.apache.hudi.table.format.cow.vector.reader;
 
+import org.apache.hudi.table.format.cow.ParquetSplitReaderUtil;
 import org.apache.hudi.table.format.cow.vector.ParquetDecimalVector;
+import org.apache.hudi.table.format.cow.vector.type.ParquetField;
 
 import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
 import org.apache.flink.table.data.RowData;
@@ -28,6 +30,7 @@ import org.apache.flink.table.data.columnar.vector.VectorizedColumnBatch;
 import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
 import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.table.types.logical.RowType;
 import org.apache.flink.util.FlinkRuntimeException;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.Path;
@@ -39,6 +42,8 @@ import org.apache.parquet.filter2.predicate.FilterPredicate;
 import org.apache.parquet.hadoop.ParquetFileReader;
 import org.apache.parquet.hadoop.metadata.BlockMetaData;
 import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.MessageColumnIO;
 import org.apache.parquet.schema.GroupType;
 import org.apache.parquet.schema.MessageType;
 import org.apache.parquet.schema.Type;
@@ -46,6 +51,7 @@ import org.apache.parquet.schema.Types;
 
 import java.io.Closeable;
 import java.io.IOException;
+import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.HashMap;
 import java.util.List;
@@ -53,7 +59,6 @@ import java.util.Locale;
 import java.util.Map;
 import java.util.stream.IntStream;
 
-import static org.apache.hudi.table.format.cow.ParquetSplitReaderUtil.createColumnReader;
 import static org.apache.hudi.table.format.cow.ParquetSplitReaderUtil.createWritableColumnVector;
 import static org.apache.parquet.filter2.compat.FilterCompat.get;
 import static org.apache.parquet.filter2.compat.RowGroupFilter.filterRowGroups;
@@ -77,6 +82,14 @@ public class ParquetColumnarRowSplitReader implements Closeable {
 
   private final MessageType requestedSchema;
 
+  /**
+   * {@link ParquetField} tree per top-level requested column, used by
+   * {@link ParquetSplitReaderUtil#createColumnReader(boolean, LogicalType, Type, List,
+   * PageReadStore, ParquetField)} to drive the Dremel-style {@link NestedColumnReader} for
+   * nested types. Entries are {@code null} for primitive top-level fields. Built once per split.
+   */
+  private final List<ParquetField> requestedFields;
+
   /**
    * The total number of rows this RecordReader will eventually read. The sum of the rows of all
    * the row groups.
@@ -158,6 +171,20 @@ public class ParquetColumnarRowSplitReader implements Closeable {
 
     checkSchema();
 
+    // Build the ParquetField tree once per split (the Dremel-style nested reader reuses it across
+    // row groups). Only columns with nested logical type get a non-null entry β€” primitive columns
+    // still use Hudi's specialized ColumnReaders.
+    MessageColumnIO messageColumnIO = new ColumnIOFactory().getColumnIO(requestedSchema);
+    List<RowType.RowField> requestedRowFields = new ArrayList<>(requestedTypes.length);
+    List<String> requestedFieldNames = new ArrayList<>(requestedTypes.length);
+    for (int i = 0; i < requestedTypes.length; i++) {
+      String name = requestedSchema.getFieldName(i);
+      requestedRowFields.add(new RowType.RowField(name, requestedTypes[i]));
+      requestedFieldNames.add(name);
+    }
+    this.requestedFields = ParquetSplitReaderUtil.buildFieldsList(
+        requestedRowFields, requestedFieldNames, messageColumnIO);
+
     this.writableVectors = createWritableVectors();
     ColumnVector[] columnVectors = patchedVector(selectedFieldNames.length, createReadableVectors(), requestedIndices);
     this.columnarBatch = generator.generate(columnVectors);
@@ -340,12 +367,13 @@ public class ParquetColumnarRowSplitReader implements Closeable {
     List<ColumnDescriptor> columns = requestedSchema.getColumns();
     columnReaders = new ColumnReader[types.size()];
     for (int i = 0; i < types.size(); ++i) {
-      columnReaders[i] = createColumnReader(
+      columnReaders[i] = ParquetSplitReaderUtil.createColumnReader(
           utcTimestamp,
           requestedTypes[i],
           types.get(i),
           columns,
-          pages);
+          pages,
+          requestedFields.get(i));
     }
     totalCountLoadedSoFar += pages.getRowCount();
   }


(hudi) 05/16: feat(flink): Backport Flink 2.1 nested Parquet column readers and INT64 timestamp dispatch (FLINK-35702) (#18636)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit b5342e756b838b9f562e7e8007eab6f4a897d524
Author: Shihuan Liu <sk...@gmail.com>
AuthorDate: Wed May 6 23:23:48 2026 -0700

    feat(flink): Backport Flink 2.1 nested Parquet column readers and INT64 timestamp dispatch (FLINK-35702) (#18636)
    
    * feat(flink): Backport Flink 2.1 nested Parquet column readers and INT64 timestamp dispatch (FLINK-35702)
    * Minor fixes
    
    (cherry picked from commit 40295605aae691da07e93a1a2919d38b41cbb6dd)
---
 .../table/format/cow/vector/HeapArrayVector.java   |  31 +
 .../format/cow/vector/HeapMapColumnVector.java     |  63 +-
 .../format/cow/vector/HeapRowColumnVector.java     |  15 +
 .../cow/vector/reader/NestedColumnReader.java      | 257 +++++++++
 .../vector/reader/NestedPrimitiveColumnReader.java | 639 +++++++++++++++++++++
 .../reader/ParquetDataColumnReaderFactory.java     | 108 +++-
 .../cow/vector/TestHeapColumnVectorAccessors.java  | 138 +++++
 .../reader/TestParquetDataColumnReaderFactory.java | 272 +++++++++
 8 files changed, 1516 insertions(+), 7 deletions(-)

diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapArrayVector.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapArrayVector.java
index a0dced01e5e8..2f21a323302f 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapArrayVector.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapArrayVector.java
@@ -57,6 +57,37 @@ public class HeapArrayVector extends AbstractHeapVector
     return this.isNull.length;
   }
 
+  // ---------------------------------------------------------------------------------------------
+  // Flink 2.1-compatible accessors. Backed by the existing public {@code offsets}, {@code lengths}
+  // and {@code child} fields so legacy callers continue to work; the new {@link
+  // org.apache.hudi.table.format.cow.vector.reader.NestedColumnReader} (FLINK-35702 port) and any
+  // future Flink-2.1-style caller use these accessors.
+  // ---------------------------------------------------------------------------------------------
+
+  public long[] getOffsets() {
+    return offsets;
+  }
+
+  public void setOffsets(long[] offsets) {
+    this.offsets = offsets;
+  }
+
+  public long[] getLengths() {
+    return lengths;
+  }
+
+  public void setLengths(long[] lengths) {
+    this.lengths = lengths;
+  }
+
+  public ColumnVector getChild() {
+    return child;
+  }
+
+  public void setChild(ColumnVector child) {
+    this.child = child;
+  }
+
   @Override
   public ArrayData getArray(int i) {
     long offset = offsets[i];
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapMapColumnVector.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapMapColumnVector.java
index 0d83f82baedf..a98bdebd707a 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapMapColumnVector.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapMapColumnVector.java
@@ -20,6 +20,7 @@ package org.apache.hudi.table.format.cow.vector;
 
 import lombok.Getter;
 import org.apache.flink.table.data.MapData;
+import org.apache.flink.table.data.columnar.vector.ColumnVector;
 import org.apache.flink.table.data.columnar.vector.MapColumnVector;
 import org.apache.flink.table.data.columnar.vector.heap.AbstractHeapVector;
 import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
@@ -31,16 +32,74 @@ public class HeapMapColumnVector extends AbstractHeapVector
     implements WritableColumnVector, MapColumnVector {
 
   @Getter
-  private final WritableColumnVector keys;
+  private WritableColumnVector keys;
   @Getter
-  private final WritableColumnVector values;
+  private WritableColumnVector values;
+
+  // ---------------------------------------------------------------------------------------------
+  // Flink 2.1 Dremel-style state. Populated by {@link
+  // org.apache.hudi.table.format.cow.vector.reader.NestedColumnReader} (FLINK-35702 port). The
+  // legacy {@link #getMap(int)} implementation below continues to use {@code ColumnarGroupMapData}
+  // β€” wiring it through these offsets/lengths happens in a follow-up PR that switches the read
+  // path. Left here so the new readers can compile against the additive surface.
+  // ---------------------------------------------------------------------------------------------
+  private long[] offsets;
+  private long[] lengths;
+  private int size;
 
   public HeapMapColumnVector(int len, WritableColumnVector keys, WritableColumnVector values) {
     super(len);
+    this.offsets = new long[len];
+    this.lengths = new long[len];
     this.keys = keys;
     this.values = values;
   }
 
+  public long[] getOffsets() {
+    return offsets;
+  }
+
+  public void setOffsets(long[] offsets) {
+    this.offsets = offsets;
+  }
+
+  public long[] getLengths() {
+    return lengths;
+  }
+
+  public void setLengths(long[] lengths) {
+    this.lengths = lengths;
+  }
+
+  public int getSize() {
+    return size;
+  }
+
+  public void setSize(int size) {
+    this.size = size;
+  }
+
+  public void setKeys(WritableColumnVector keys) {
+    this.keys = keys;
+  }
+
+  public void setValues(WritableColumnVector values) {
+    this.values = values;
+  }
+
+  /**
+   * Returns the keys child vector typed as {@link ColumnVector}, matching the Flink 2.1 contract
+   * consumed by {@code NestedColumnReader}. Functionally equivalent to {@link #getKeys()}.
+   */
+  public ColumnVector getKeyColumnVector() {
+    return keys;
+  }
+
+  /** Counterpart of {@link #getKeyColumnVector()} for the values child vector. */
+  public ColumnVector getValueColumnVector() {
+    return values;
+  }
+
   @Override
   public MapData getMap(int rowId) {
     return new ColumnarGroupMapData(keys, values, rowId);
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapRowColumnVector.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapRowColumnVector.java
index ae194e4e6ab0..0c640ce92ee4 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapRowColumnVector.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapRowColumnVector.java
@@ -37,6 +37,21 @@ public class HeapRowColumnVector extends AbstractHeapVector
     this.vectors = vectors;
   }
 
+  /**
+   * Flink 2.1-compatible accessor for the children vectors. Backed by the existing public {@code
+   * vectors} field so legacy callers continue to work; the new {@link
+   * org.apache.hudi.table.format.cow.vector.reader.NestedColumnReader} (FLINK-35702 port) and any
+   * future Flink-2.1-style caller use this accessor.
+   */
+  public WritableColumnVector[] getFields() {
+    return vectors;
+  }
+
+  /** Counterpart of {@link #getFields()}. */
+  public void setFields(WritableColumnVector[] fields) {
+    this.vectors = fields;
+  }
+
   @Override
   public ColumnarRowData getRow(int i) {
     ColumnarRowData columnarRowData = new ColumnarRowData(new VectorizedColumnBatch(vectors));
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java
new file mode 100644
index 000000000000..27eab298b320
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java
@@ -0,0 +1,257 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.format.cow.vector.reader;
+
+import org.apache.hudi.table.format.cow.utils.NestedPositionUtil;
+import org.apache.hudi.table.format.cow.vector.HeapArrayVector;
+import org.apache.hudi.table.format.cow.vector.HeapMapColumnVector;
+import org.apache.hudi.table.format.cow.vector.HeapRowColumnVector;
+import org.apache.hudi.table.format.cow.vector.position.CollectionPosition;
+import org.apache.hudi.table.format.cow.vector.position.LevelDelegation;
+import org.apache.hudi.table.format.cow.vector.position.RowPosition;
+import org.apache.hudi.table.format.cow.vector.type.ParquetField;
+import org.apache.hudi.table.format.cow.vector.type.ParquetGroupField;
+import org.apache.hudi.table.format.cow.vector.type.ParquetPrimitiveField;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
+import org.apache.flink.table.data.columnar.vector.ColumnVector;
+import org.apache.flink.table.data.columnar.vector.heap.AbstractHeapVector;
+import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
+import org.apache.flink.table.types.logical.ArrayType;
+import org.apache.flink.table.types.logical.MapType;
+import org.apache.flink.table.types.logical.MultisetType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.page.PageReadStore;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * ColumnReader used to read a {@code Group} type in Parquet ({@code Map}, {@code Array}, {@code
+ * Row}). Resolves nested structures using Dremel striping/assembly; see <a
+ * href="https://github.com/julienledem/redelm/wiki/The-striping-and-assembly-algorithms-from-the-Dremel-paper">the
+ * striping and assembly algorithms from the Dremel paper</a>.
+ *
+ * <p>Vendored from Apache Flink 2.1 (FLINK-35702, {@code
+ * org.apache.flink.formats.parquet.vector.reader.NestedColumnReader}). Differences vs. upstream:
+ *
+ * <ul>
+ *   <li>Uses Hudi-local {@code HeapRowColumnVector}/{@code HeapMapColumnVector}/{@code
+ *       HeapArrayVector} instead of the Flink-private {@code HeapRowVector}/{@code
+ *       HeapMapVector}/{@code HeapArrayVector}.
+ *   <li>Supports Hudi's schema-evolution contract: a {@code ParquetGroupField} representing a
+ *       {@link RowType} may contain {@code null} children β€” meaning the corresponding logical
+ *       field is absent from the Parquet file. Those slots are passed through unchanged and do
+ *       not contribute to the row's repetition/definition-level stream.
+ * </ul>
+ */
+public class NestedColumnReader implements ColumnReader<WritableColumnVector> {
+
+  private final Map<ColumnDescriptor, NestedPrimitiveColumnReader> columnReaders;
+  private final boolean isUtcTimestamp;
+
+  private final PageReadStore pages;
+
+  private final ParquetField field;
+
+  public NestedColumnReader(boolean isUtcTimestamp, PageReadStore pages, ParquetField field) {
+    this.isUtcTimestamp = isUtcTimestamp;
+    this.pages = pages;
+    this.field = field;
+    this.columnReaders = new HashMap<>();
+  }
+
+  @Override
+  public void readToVector(int readNumber, WritableColumnVector vector) throws IOException {
+    readData(field, readNumber, vector, false);
+  }
+
+  private Tuple2<LevelDelegation, WritableColumnVector> readData(
+      ParquetField field, int readNumber, ColumnVector vector, boolean inside) throws IOException {
+    if (field.getType() instanceof RowType) {
+      return readRow((ParquetGroupField) field, readNumber, vector, inside);
+    } else if (field.getType() instanceof MapType || field.getType() instanceof MultisetType) {
+      return readMap((ParquetGroupField) field, readNumber, vector, inside);
+    } else if (field.getType() instanceof ArrayType) {
+      return readArray((ParquetGroupField) field, readNumber, vector, inside);
+    } else {
+      return readPrimitive((ParquetPrimitiveField) field, readNumber, vector);
+    }
+  }
+
+  private Tuple2<LevelDelegation, WritableColumnVector> readRow(
+      ParquetGroupField field, int readNumber, ColumnVector vector, boolean inside)
+      throws IOException {
+    HeapRowColumnVector heapRowVector = (HeapRowColumnVector) vector;
+    LevelDelegation levelDelegation = null;
+    List<ParquetField> children = field.getChildren();
+    WritableColumnVector[] childrenVectors = heapRowVector.getFields();
+    WritableColumnVector[] finalChildrenVectors = new WritableColumnVector[childrenVectors.length];
+    for (int i = 0; i < children.size(); i++) {
+      ParquetField child = children.get(i);
+      if (child == null) {
+        // Hudi schema-evolution: the logical field is not present in the Parquet file. The slot
+        // vector is expected to be pre-populated with nulls by the caller (lands in a follow-up
+        // PR that rewires ParquetSplitReaderUtil); keep it as is and skip contributing to the
+        // level stream.
+        finalChildrenVectors[i] = childrenVectors[i];
+        continue;
+      }
+      Tuple2<LevelDelegation, WritableColumnVector> tuple =
+          readData(child, readNumber, childrenVectors[i], true);
+      levelDelegation = tuple.f0;
+      finalChildrenVectors[i] = tuple.f1;
+    }
+    if (levelDelegation == null) {
+      throw new FlinkRuntimeException(
+          String.format("Row field does not have any non-null children: %s.", field));
+    }
+
+    RowPosition rowPosition =
+        NestedPositionUtil.calculateRowOffsets(
+            field,
+            levelDelegation.getDefinitionLevel(),
+            levelDelegation.getRepetitionLevel());
+
+    // If row was inside the structure, then we need to renew the vector to reset the
+    // capacity.
+    if (inside) {
+      heapRowVector = new HeapRowColumnVector(rowPosition.getPositionsCount(), finalChildrenVectors);
+    } else {
+      heapRowVector.setFields(finalChildrenVectors);
+    }
+
+    if (rowPosition.getIsNull() != null) {
+      setFieldNullFlag(rowPosition.getIsNull(), heapRowVector);
+    }
+    return Tuple2.of(levelDelegation, heapRowVector);
+  }
+
+  private Tuple2<LevelDelegation, WritableColumnVector> readMap(
+      ParquetGroupField field, int readNumber, ColumnVector vector, boolean inside)
+      throws IOException {
+    HeapMapColumnVector mapVector = (HeapMapColumnVector) vector;
+    mapVector.reset();
+    List<ParquetField> children = field.getChildren();
+    Preconditions.checkArgument(
+        children.size() == 2,
+        "Maps must have two type parameters, found %s",
+        children.size());
+    Tuple2<LevelDelegation, WritableColumnVector> keyTuple =
+        readData(children.get(0), readNumber, mapVector.getKeyColumnVector(), true);
+    Tuple2<LevelDelegation, WritableColumnVector> valueTuple =
+        readData(children.get(1), readNumber, mapVector.getValueColumnVector(), true);
+
+    LevelDelegation levelDelegation = keyTuple.f0;
+
+    CollectionPosition collectionPosition =
+        NestedPositionUtil.calculateCollectionOffsets(
+            field,
+            levelDelegation.getDefinitionLevel(),
+            levelDelegation.getRepetitionLevel());
+
+    // If map was inside the structure, then we need to renew the vector to reset the
+    // capacity.
+    if (inside) {
+      mapVector = new HeapMapColumnVector(collectionPosition.getValueCount(), keyTuple.f1, valueTuple.f1);
+    } else {
+      mapVector.setKeys(keyTuple.f1);
+      mapVector.setValues(valueTuple.f1);
+    }
+
+    if (collectionPosition.getIsNull() != null) {
+      setFieldNullFlag(collectionPosition.getIsNull(), mapVector);
+    }
+
+    mapVector.setLengths(collectionPosition.getLength());
+    mapVector.setOffsets(collectionPosition.getOffsets());
+
+    return Tuple2.of(levelDelegation, mapVector);
+  }
+
+  private Tuple2<LevelDelegation, WritableColumnVector> readArray(
+      ParquetGroupField field, int readNumber, ColumnVector vector, boolean inside)
+      throws IOException {
+    HeapArrayVector arrayVector = (HeapArrayVector) vector;
+    arrayVector.reset();
+    List<ParquetField> children = field.getChildren();
+    Preconditions.checkArgument(
+        children.size() == 1,
+        "Arrays must have a single type parameter, found %s",
+        children.size());
+    Tuple2<LevelDelegation, WritableColumnVector> tuple =
+        readData(children.get(0), readNumber, arrayVector.getChild(), true);
+
+    LevelDelegation levelDelegation = tuple.f0;
+    CollectionPosition collectionPosition =
+        NestedPositionUtil.calculateCollectionOffsets(
+            field,
+            levelDelegation.getDefinitionLevel(),
+            levelDelegation.getRepetitionLevel());
+
+    // If array was inside the structure, then we need to renew the vector to reset the
+    // capacity.
+    if (inside) {
+      arrayVector = new HeapArrayVector(collectionPosition.getValueCount(), tuple.f1);
+    } else {
+      arrayVector.setChild(tuple.f1);
+    }
+
+    if (collectionPosition.getIsNull() != null) {
+      setFieldNullFlag(collectionPosition.getIsNull(), arrayVector);
+    }
+    arrayVector.setLengths(collectionPosition.getLength());
+    arrayVector.setOffsets(collectionPosition.getOffsets());
+    return Tuple2.of(levelDelegation, arrayVector);
+  }
+
+  private Tuple2<LevelDelegation, WritableColumnVector> readPrimitive(
+      ParquetPrimitiveField field, int readNumber, ColumnVector vector) throws IOException {
+    ColumnDescriptor descriptor = field.getDescriptor();
+    NestedPrimitiveColumnReader reader = columnReaders.get(descriptor);
+    if (reader == null) {
+      reader =
+          new NestedPrimitiveColumnReader(
+              descriptor,
+              pages.getPageReader(descriptor),
+              isUtcTimestamp,
+              descriptor.getPrimitiveType(),
+              field.getType());
+      columnReaders.put(descriptor, reader);
+    }
+    WritableColumnVector writableColumnVector =
+        reader.readAndNewVector(readNumber, (WritableColumnVector) vector);
+    return Tuple2.of(reader.getLevelDelegation(), writableColumnVector);
+  }
+
+  private static void setFieldNullFlag(boolean[] nullFlags, AbstractHeapVector vector) {
+    for (int index = 0; index < vector.getLen() && index < nullFlags.length; index++) {
+      if (nullFlags[index]) {
+        vector.setNullAt(index);
+      }
+    }
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedPrimitiveColumnReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedPrimitiveColumnReader.java
new file mode 100644
index 000000000000..3f0aefe2af74
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedPrimitiveColumnReader.java
@@ -0,0 +1,639 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.format.cow.vector.reader;
+
+import org.apache.hudi.table.format.cow.utils.IntArrayList;
+import org.apache.hudi.table.format.cow.vector.ParquetDecimalVector;
+import org.apache.hudi.table.format.cow.vector.position.LevelDelegation;
+
+import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
+import org.apache.flink.table.data.TimestampData;
+import org.apache.flink.table.data.columnar.vector.heap.HeapBooleanVector;
+import org.apache.flink.table.data.columnar.vector.heap.HeapByteVector;
+import org.apache.flink.table.data.columnar.vector.heap.HeapBytesVector;
+import org.apache.flink.table.data.columnar.vector.heap.HeapDoubleVector;
+import org.apache.flink.table.data.columnar.vector.heap.HeapFloatVector;
+import org.apache.flink.table.data.columnar.vector.heap.HeapIntVector;
+import org.apache.flink.table.data.columnar.vector.heap.HeapLongVector;
+import org.apache.flink.table.data.columnar.vector.heap.HeapShortVector;
+import org.apache.flink.table.data.columnar.vector.heap.HeapTimestampVector;
+import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
+import org.apache.flink.table.types.logical.LogicalType;
+
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.page.DataPage;
+import org.apache.parquet.column.page.DataPageV1;
+import org.apache.parquet.column.page.DataPageV2;
+import org.apache.parquet.column.page.DictionaryPage;
+import org.apache.parquet.column.page.PageReader;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.column.values.rle.RunLengthBitPackingHybridDecoder;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.parquet.column.ValuesType.DEFINITION_LEVEL;
+import static org.apache.parquet.column.ValuesType.REPETITION_LEVEL;
+import static org.apache.parquet.column.ValuesType.VALUES;
+
+/**
+ * Reader to read a single primitive leaf column that participates in a nested (Dremel) structure.
+ *
+ * <p>Vendored from Apache Flink 2.1 (FLINK-35702, {@code
+ * org.apache.flink.formats.parquet.vector.reader.NestedPrimitiveColumnReader}). Only the package
+ * and the Hudi-local {@link ParquetDecimalVector} / {@link LevelDelegation} / {@link IntArrayList}
+ * imports are changed; the algorithm is untouched. The companion Hudi-specific {@code
+ * Int64TimestampColumnReader} / {@code FixedLenBytesColumnReader} behaviours stay at the leaf-
+ * reader creation boundary in {@code ParquetSplitReaderUtil} (lands in a follow-up PR), not inside
+ * this class β€” keeping it a faithful copy of upstream.
+ */
+public class NestedPrimitiveColumnReader implements ColumnReader<WritableColumnVector> {
+  private static final Logger LOG = LoggerFactory.getLogger(NestedPrimitiveColumnReader.class);
+
+  private final IntArrayList repetitionLevelList = new IntArrayList(0);
+  private final IntArrayList definitionLevelList = new IntArrayList(0);
+
+  private final PageReader pageReader;
+  private final ColumnDescriptor descriptor;
+  private final Type type;
+  private final LogicalType logicalType;
+
+  /** The dictionary, if this column has dictionary encoding. */
+  private final ParquetDataColumnReader dictionary;
+
+  /** Maximum definition level for this column. */
+  private final int maxDefLevel;
+
+  private boolean isUtcTimestamp;
+
+  /** Total number of values read. */
+  private long valuesRead;
+
+  /**
+   * value that indicates the end of the current page. That is, if valuesRead ==
+   * endOfPageValueCount, we are at the end of the page.
+   */
+  private long endOfPageValueCount;
+
+  /** If true, the current page is dictionary encoded. */
+  private boolean isCurrentPageDictionaryEncoded;
+
+  private int definitionLevel;
+  private int repetitionLevel;
+
+  /** Repetition/Definition/Value readers. */
+  private IntIterator repetitionLevelColumn;
+
+  private IntIterator definitionLevelColumn;
+  private ParquetDataColumnReader dataColumn;
+
+  /** Total values in the current page. */
+  private int pageValueCount;
+
+  // flag to indicate if there is no data in parquet data page
+  private boolean eof = false;
+
+  private boolean isFirstRow = true;
+
+  private Object lastValue;
+
+  public NestedPrimitiveColumnReader(
+      ColumnDescriptor descriptor,
+      PageReader pageReader,
+      boolean isUtcTimestamp,
+      Type parquetType,
+      LogicalType logicalType)
+      throws IOException {
+    this.descriptor = descriptor;
+    this.type = parquetType;
+    this.pageReader = pageReader;
+    this.maxDefLevel = descriptor.getMaxDefinitionLevel();
+    this.isUtcTimestamp = isUtcTimestamp;
+    this.logicalType = logicalType;
+
+    DictionaryPage dictionaryPage = pageReader.readDictionaryPage();
+    if (dictionaryPage != null) {
+      try {
+        this.dictionary =
+            ParquetDataColumnReaderFactory.getDataColumnReaderByTypeOnDictionary(
+                parquetType.asPrimitiveType(),
+                dictionaryPage.getEncoding().initDictionary(descriptor, dictionaryPage),
+                isUtcTimestamp);
+        this.isCurrentPageDictionaryEncoded = true;
+      } catch (IOException e) {
+        throw new IOException(
+            String.format("Could not decode the dictionary for %s", descriptor), e);
+      }
+    } else {
+      this.dictionary = null;
+      this.isCurrentPageDictionaryEncoded = false;
+    }
+  }
+
+  // Not invoked directly; callers use readAndNewVector instead.
+  @Override
+  public void readToVector(int readNumber, WritableColumnVector vector) throws IOException {
+    throw new UnsupportedOperationException("This function should not be called.");
+  }
+
+  public WritableColumnVector readAndNewVector(int readNumber, WritableColumnVector vector)
+      throws IOException {
+    if (isFirstRow) {
+      if (!readValue()) {
+        return vector;
+      }
+      isFirstRow = false;
+    }
+
+    // index to set value.
+    int index = 0;
+    int valueIndex = 0;
+    List<Object> valueList = new ArrayList<>();
+
+    // repeated type need two loops to read data.
+    while (!eof && index < readNumber) {
+      do {
+        valueList.add(lastValue);
+        valueIndex++;
+      } while (readValue() && (repetitionLevel != 0));
+      index++;
+    }
+
+    return fillColumnVector(valueIndex, valueList);
+  }
+
+  public LevelDelegation getLevelDelegation() {
+    int[] repetition = repetitionLevelList.toArray();
+    int[] definition = definitionLevelList.toArray();
+    repetitionLevelList.clear();
+    definitionLevelList.clear();
+    repetitionLevelList.add(repetitionLevel);
+    definitionLevelList.add(definitionLevel);
+    return new LevelDelegation(repetition, definition);
+  }
+
+  private boolean readValue() throws IOException {
+    int left = readPageIfNeed();
+    if (left > 0) {
+      // get the values of repetition and definitionLevel
+      readAndSaveRepetitionAndDefinitionLevels();
+      // read the data if it isn't null
+      if (definitionLevel == maxDefLevel) {
+        if (isCurrentPageDictionaryEncoded) {
+          int dictionaryId = dataColumn.readValueDictionaryId();
+          lastValue = dictionaryDecodeValue(logicalType, dictionaryId);
+        } else {
+          lastValue = readPrimitiveTypedRow(logicalType);
+        }
+      } else {
+        lastValue = null;
+      }
+      return true;
+    } else {
+      eof = true;
+      return false;
+    }
+  }
+
+  private void readAndSaveRepetitionAndDefinitionLevels() {
+    // get the values of repetition and definitionLevel
+    repetitionLevel = repetitionLevelColumn.nextInt();
+    definitionLevel = definitionLevelColumn.nextInt();
+    valuesRead++;
+    repetitionLevelList.add(repetitionLevel);
+    definitionLevelList.add(definitionLevel);
+  }
+
+  private int readPageIfNeed() throws IOException {
+    // Compute the number of values we want to read in this page.
+    int leftInPage = (int) (endOfPageValueCount - valuesRead);
+    if (leftInPage == 0) {
+      // no data left in current page, load data from new page
+      readPage();
+      leftInPage = (int) (endOfPageValueCount - valuesRead);
+    }
+    return leftInPage;
+  }
+
+  private Object readPrimitiveTypedRow(LogicalType category) {
+    switch (category.getTypeRoot()) {
+      case CHAR:
+      case VARCHAR:
+      case BINARY:
+      case VARBINARY:
+        return dataColumn.readBytes();
+      case BOOLEAN:
+        return dataColumn.readBoolean();
+      case TIME_WITHOUT_TIME_ZONE:
+      case DATE:
+      case INTEGER:
+        return dataColumn.readInteger();
+      case TINYINT:
+        return dataColumn.readTinyInt();
+      case SMALLINT:
+        return dataColumn.readSmallInt();
+      case BIGINT:
+        return dataColumn.readLong();
+      case FLOAT:
+        return dataColumn.readFloat();
+      case DOUBLE:
+        return dataColumn.readDouble();
+      case DECIMAL:
+        switch (descriptor.getPrimitiveType().getPrimitiveTypeName()) {
+          case INT32:
+            return dataColumn.readInteger();
+          case INT64:
+            return dataColumn.readLong();
+          case BINARY:
+          case FIXED_LEN_BYTE_ARRAY:
+            return dataColumn.readBytes();
+          default:
+            throw new RuntimeException(
+                "Unsupported physical type for DECIMAL: " + descriptor.getPrimitiveType());
+        }
+      case TIMESTAMP_WITHOUT_TIME_ZONE:
+      case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+        return dataColumn.readTimestamp();
+      default:
+        throw new RuntimeException("Unsupported type in the list: " + type);
+    }
+  }
+
+  private Object dictionaryDecodeValue(LogicalType category, Integer dictionaryValue) {
+    if (dictionaryValue == null) {
+      return null;
+    }
+
+    switch (category.getTypeRoot()) {
+      case CHAR:
+      case VARCHAR:
+      case BINARY:
+      case VARBINARY:
+        return dictionary.readBytes(dictionaryValue);
+      case DATE:
+      case TIME_WITHOUT_TIME_ZONE:
+      case INTEGER:
+        return dictionary.readInteger(dictionaryValue);
+      case BOOLEAN:
+        return dictionary.readBoolean(dictionaryValue) ? 1 : 0;
+      case DOUBLE:
+        return dictionary.readDouble(dictionaryValue);
+      case FLOAT:
+        return dictionary.readFloat(dictionaryValue);
+      case TINYINT:
+        return dictionary.readTinyInt(dictionaryValue);
+      case SMALLINT:
+        return dictionary.readSmallInt(dictionaryValue);
+      case BIGINT:
+        return dictionary.readLong(dictionaryValue);
+      case DECIMAL:
+        switch (descriptor.getPrimitiveType().getPrimitiveTypeName()) {
+          case INT32:
+            return dictionary.readInteger(dictionaryValue);
+          case INT64:
+            return dictionary.readLong(dictionaryValue);
+          case FIXED_LEN_BYTE_ARRAY:
+          case BINARY:
+            return dictionary.readBytes(dictionaryValue);
+          default:
+            throw new RuntimeException(
+                "Unsupported physical type for DECIMAL: " + descriptor.getPrimitiveType());
+        }
+      case TIMESTAMP_WITHOUT_TIME_ZONE:
+      case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+        return dictionary.readTimestamp(dictionaryValue);
+      default:
+        throw new RuntimeException("Unsupported type in the list: " + type);
+    }
+  }
+
+  @SuppressWarnings({"unchecked", "rawtypes"})
+  private WritableColumnVector fillColumnVector(int total, List valueList) {
+    switch (logicalType.getTypeRoot()) {
+      case CHAR:
+      case VARCHAR:
+      case BINARY:
+      case VARBINARY:
+        HeapBytesVector heapBytesVector = new HeapBytesVector(total);
+        for (int i = 0; i < valueList.size(); i++) {
+          byte[] src = ((List<byte[]>) valueList).get(i);
+          if (src == null) {
+            heapBytesVector.setNullAt(i);
+          } else {
+            heapBytesVector.appendBytes(i, src, 0, src.length);
+          }
+        }
+        return heapBytesVector;
+      case BOOLEAN:
+        HeapBooleanVector heapBooleanVector = new HeapBooleanVector(total);
+        for (int i = 0; i < valueList.size(); i++) {
+          if (valueList.get(i) == null) {
+            heapBooleanVector.setNullAt(i);
+          } else {
+            heapBooleanVector.vector[i] = ((List<Boolean>) valueList).get(i);
+          }
+        }
+        return heapBooleanVector;
+      case TINYINT:
+        HeapByteVector heapByteVector = new HeapByteVector(total);
+        for (int i = 0; i < valueList.size(); i++) {
+          if (valueList.get(i) == null) {
+            heapByteVector.setNullAt(i);
+          } else {
+            heapByteVector.vector[i] = (byte) ((List<Integer>) valueList).get(i).intValue();
+          }
+        }
+        return heapByteVector;
+      case SMALLINT:
+        HeapShortVector heapShortVector = new HeapShortVector(total);
+        for (int i = 0; i < valueList.size(); i++) {
+          if (valueList.get(i) == null) {
+            heapShortVector.setNullAt(i);
+          } else {
+            heapShortVector.vector[i] = (short) ((List<Integer>) valueList).get(i).intValue();
+          }
+        }
+        return heapShortVector;
+      case INTEGER:
+      case DATE:
+      case TIME_WITHOUT_TIME_ZONE:
+        HeapIntVector heapIntVector = new HeapIntVector(total);
+        for (int i = 0; i < valueList.size(); i++) {
+          if (valueList.get(i) == null) {
+            heapIntVector.setNullAt(i);
+          } else {
+            heapIntVector.vector[i] = ((List<Integer>) valueList).get(i);
+          }
+        }
+        return heapIntVector;
+      case FLOAT:
+        HeapFloatVector heapFloatVector = new HeapFloatVector(total);
+        for (int i = 0; i < valueList.size(); i++) {
+          if (valueList.get(i) == null) {
+            heapFloatVector.setNullAt(i);
+          } else {
+            heapFloatVector.vector[i] = ((List<Float>) valueList).get(i);
+          }
+        }
+        return heapFloatVector;
+      case BIGINT:
+        HeapLongVector heapLongVector = new HeapLongVector(total);
+        for (int i = 0; i < valueList.size(); i++) {
+          if (valueList.get(i) == null) {
+            heapLongVector.setNullAt(i);
+          } else {
+            heapLongVector.vector[i] = ((List<Long>) valueList).get(i);
+          }
+        }
+        return heapLongVector;
+      case DOUBLE:
+        HeapDoubleVector heapDoubleVector = new HeapDoubleVector(total);
+        for (int i = 0; i < valueList.size(); i++) {
+          if (valueList.get(i) == null) {
+            heapDoubleVector.setNullAt(i);
+          } else {
+            heapDoubleVector.vector[i] = ((List<Double>) valueList).get(i);
+          }
+        }
+        return heapDoubleVector;
+      case TIMESTAMP_WITHOUT_TIME_ZONE:
+      case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
+        HeapTimestampVector heapTimestampVector = new HeapTimestampVector(total);
+        for (int i = 0; i < valueList.size(); i++) {
+          if (valueList.get(i) == null) {
+            heapTimestampVector.setNullAt(i);
+          } else {
+            heapTimestampVector.setTimestamp(i, ((List<TimestampData>) valueList).get(i));
+          }
+        }
+        return heapTimestampVector;
+      case DECIMAL:
+        PrimitiveType.PrimitiveTypeName primitiveTypeName =
+            descriptor.getPrimitiveType().getPrimitiveTypeName();
+        switch (primitiveTypeName) {
+          case INT32:
+            HeapIntVector phiv = new HeapIntVector(total);
+            for (int i = 0; i < valueList.size(); i++) {
+              if (valueList.get(i) == null) {
+                phiv.setNullAt(i);
+              } else {
+                phiv.vector[i] = ((List<Integer>) valueList).get(i);
+              }
+            }
+            return new ParquetDecimalVector(phiv);
+          case INT64:
+            HeapLongVector phlv = new HeapLongVector(total);
+            for (int i = 0; i < valueList.size(); i++) {
+              if (valueList.get(i) == null) {
+                phlv.setNullAt(i);
+              } else {
+                phlv.vector[i] = ((List<Long>) valueList).get(i);
+              }
+            }
+            return new ParquetDecimalVector(phlv);
+          default:
+            HeapBytesVector phbv = getHeapBytesVector(total, valueList);
+            return new ParquetDecimalVector(phbv);
+        }
+      default:
+        throw new RuntimeException("Unsupported type in the list: " + type);
+    }
+  }
+
+  @SuppressWarnings({"unchecked", "rawtypes"})
+  private static HeapBytesVector getHeapBytesVector(int total, List valueList) {
+    HeapBytesVector phbv = new HeapBytesVector(total);
+    for (int i = 0; i < valueList.size(); i++) {
+      byte[] src = ((List<byte[]>) valueList).get(i);
+      if (valueList.get(i) == null) {
+        phbv.setNullAt(i);
+      } else {
+        phbv.appendBytes(i, src, 0, src.length);
+      }
+    }
+    return phbv;
+  }
+
+  protected void readPage() {
+    DataPage page = pageReader.readPage();
+
+    if (page == null) {
+      return;
+    }
+
+    page.accept(
+        new DataPage.Visitor<Void>() {
+          @Override
+          public Void visit(DataPageV1 dataPageV1) {
+            readPageV1(dataPageV1);
+            return null;
+          }
+
+          @Override
+          public Void visit(DataPageV2 dataPageV2) {
+            readPageV2(dataPageV2);
+            return null;
+          }
+        });
+  }
+
+  private void initDataReader(Encoding dataEncoding, ByteBufferInputStream in, int valueCount)
+      throws IOException {
+    this.pageValueCount = valueCount;
+    this.endOfPageValueCount = valuesRead + pageValueCount;
+    if (dataEncoding.usesDictionary()) {
+      this.dataColumn = null;
+      if (dictionary == null) {
+        throw new IOException(
+            String.format(
+                "Could not read page in col %s because the dictionary was missing for encoding %s.",
+                descriptor, dataEncoding));
+      }
+      dataColumn =
+          ParquetDataColumnReaderFactory.getDataColumnReaderByType(
+              type.asPrimitiveType(),
+              dataEncoding.getDictionaryBasedValuesReader(
+                  descriptor, VALUES, dictionary.getDictionary()),
+              isUtcTimestamp);
+      this.isCurrentPageDictionaryEncoded = true;
+    } else {
+      dataColumn =
+          ParquetDataColumnReaderFactory.getDataColumnReaderByType(
+              type.asPrimitiveType(),
+              dataEncoding.getValuesReader(descriptor, VALUES),
+              isUtcTimestamp);
+      this.isCurrentPageDictionaryEncoded = false;
+    }
+
+    try {
+      dataColumn.initFromPage(pageValueCount, in);
+    } catch (IOException e) {
+      throw new IOException(String.format("Could not read page in col %s.", descriptor), e);
+    }
+  }
+
+  private void readPageV1(DataPageV1 page) {
+    ValuesReader rlReader = page.getRlEncoding().getValuesReader(descriptor, REPETITION_LEVEL);
+    ValuesReader dlReader = page.getDlEncoding().getValuesReader(descriptor, DEFINITION_LEVEL);
+    this.repetitionLevelColumn = new ValuesReaderIntIterator(rlReader);
+    this.definitionLevelColumn = new ValuesReaderIntIterator(dlReader);
+    try {
+      BytesInput bytes = page.getBytes();
+      LOG.debug("Page size {}  bytes and {} records.", bytes.size(), pageValueCount);
+      ByteBufferInputStream in = bytes.toInputStream();
+      LOG.debug("Reading repetition levels at {}.", in.position());
+      rlReader.initFromPage(pageValueCount, in);
+      LOG.debug("Reading definition levels at {}.", in.position());
+      dlReader.initFromPage(pageValueCount, in);
+      LOG.debug("Reading data at {}.", in.position());
+      initDataReader(page.getValueEncoding(), in, page.getValueCount());
+    } catch (IOException e) {
+      throw new ParquetDecodingException(
+          String.format("Could not read page %s in col %s.", page, descriptor), e);
+    }
+  }
+
+  private void readPageV2(DataPageV2 page) {
+    this.pageValueCount = page.getValueCount();
+    this.repetitionLevelColumn =
+        newRLEIterator(descriptor.getMaxRepetitionLevel(), page.getRepetitionLevels());
+    this.definitionLevelColumn =
+        newRLEIterator(descriptor.getMaxDefinitionLevel(), page.getDefinitionLevels());
+    try {
+      LOG.debug(
+          "Page data size {} bytes and {} records.", page.getData().size(), pageValueCount);
+      initDataReader(
+          page.getDataEncoding(), page.getData().toInputStream(), page.getValueCount());
+    } catch (IOException e) {
+      throw new ParquetDecodingException(
+          String.format("Could not read page %s in col %s.", page, descriptor), e);
+    }
+  }
+
+  private IntIterator newRLEIterator(int maxLevel, BytesInput bytes) {
+    try {
+      if (maxLevel == 0) {
+        return new NullIntIterator();
+      }
+      return new RLEIntIterator(
+          new RunLengthBitPackingHybridDecoder(
+              BytesUtils.getWidthFromMaxInt(maxLevel),
+              new ByteArrayInputStream(bytes.toByteArray())));
+    } catch (IOException e) {
+      throw new ParquetDecodingException(
+          String.format("Could not read levels in page for col %s.", descriptor), e);
+    }
+  }
+
+  /** Utility interface to abstract over different way to read ints with different encodings. */
+  interface IntIterator {
+    int nextInt();
+  }
+
+  /** Reading int from {@link ValuesReader}. */
+  protected static final class ValuesReaderIntIterator implements IntIterator {
+    ValuesReader delegate;
+
+    public ValuesReaderIntIterator(ValuesReader delegate) {
+      this.delegate = delegate;
+    }
+
+    @Override
+    public int nextInt() {
+      return delegate.readInteger();
+    }
+  }
+
+  /** Reading int from {@link RunLengthBitPackingHybridDecoder}. */
+  protected static final class RLEIntIterator implements IntIterator {
+    RunLengthBitPackingHybridDecoder delegate;
+
+    public RLEIntIterator(RunLengthBitPackingHybridDecoder delegate) {
+      this.delegate = delegate;
+    }
+
+    @Override
+    public int nextInt() {
+      try {
+        return delegate.readInt();
+      } catch (IOException e) {
+        throw new ParquetDecodingException(e);
+      }
+    }
+  }
+
+  /** Reading zero always. */
+  protected static final class NullIntIterator implements IntIterator {
+    @Override
+    public int nextInt() {
+      return 0;
+    }
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ParquetDataColumnReaderFactory.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ParquetDataColumnReaderFactory.java
index fdfe5d6fa3a3..3748e53fc284 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ParquetDataColumnReaderFactory.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ParquetDataColumnReaderFactory.java
@@ -26,12 +26,16 @@ import org.apache.parquet.bytes.ByteBufferInputStream;
 import org.apache.parquet.column.Dictionary;
 import org.apache.parquet.column.values.ValuesReader;
 import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.OriginalType;
 import org.apache.parquet.schema.PrimitiveType;
 
 import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
 import java.sql.Timestamp;
+import java.time.Instant;
+import java.time.temporal.ChronoUnit;
 
 import static org.apache.flink.formats.parquet.vector.reader.TimestampColumnReader.JULIAN_EPOCH_OFFSET_DAYS;
 import static org.apache.flink.formats.parquet.vector.reader.TimestampColumnReader.MILLIS_IN_DAY;
@@ -252,21 +256,115 @@ public final class ParquetDataColumnReaderFactory {
     }
   }
 
+  /**
+   * Reader for Parquet INT64 timestamp values (MILLIS / MICROS / NANOS), i.e. the standard
+   * timestamp encoding defined by Parquet's
+   * {@link LogicalTypeAnnotation.TimestampLogicalTypeAnnotation} and the legacy
+   * {@link OriginalType#TIMESTAMP_MILLIS} / {@link OriginalType#TIMESTAMP_MICROS} annotations.
+   * (The older INT96 encoding is marked deprecated by the Parquet format spec β€” see
+   * <a href="https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#timestamp">
+   * LogicalTypes.md</a> β€” but is still supported here via {@link TypesFromInt96PageReader} for
+   * backwards compatibility with files written by older Hive / Spark / Impala versions.)
+   *
+   * <p>Used by {@link NestedPrimitiveColumnReader} when a TIMESTAMP column sits inside a
+   * {@code Row}, {@code Array} or {@code Map}; the top-level path continues to use
+   * {@link Int64TimestampColumnReader} for batched-vector efficiency.
+   */
+  public static class TypesFromInt64PageReader extends DefaultParquetDataColumnReader {
+    private final boolean isUtcTimestamp;
+    private final ChronoUnit chronoUnit;
+
+    public TypesFromInt64PageReader(
+        ValuesReader realReader, boolean isUtcTimestamp, ChronoUnit chronoUnit) {
+      super(realReader);
+      this.isUtcTimestamp = isUtcTimestamp;
+      this.chronoUnit = chronoUnit;
+    }
+
+    public TypesFromInt64PageReader(
+        Dictionary dict, boolean isUtcTimestamp, ChronoUnit chronoUnit) {
+      super(dict);
+      this.isUtcTimestamp = isUtcTimestamp;
+      this.chronoUnit = chronoUnit;
+    }
+
+    @Override
+    public TimestampData readTimestamp() {
+      return int64ToTimestamp(isUtcTimestamp, valuesReader.readLong(), chronoUnit);
+    }
+
+    @Override
+    public TimestampData readTimestamp(int id) {
+      return int64ToTimestamp(isUtcTimestamp, dict.decodeToLong(id), chronoUnit);
+    }
+  }
+
   private static ParquetDataColumnReader getDataColumnReaderByTypeHelper(
       boolean isDictionary,
       PrimitiveType parquetType,
       Dictionary dictionary,
       ValuesReader valuesReader,
       boolean isUtcTimestamp) {
-    if (parquetType.getPrimitiveTypeName() == PrimitiveType.PrimitiveTypeName.INT96) {
+    PrimitiveType.PrimitiveTypeName typeName = parquetType.getPrimitiveTypeName();
+    if (typeName == PrimitiveType.PrimitiveTypeName.INT96) {
       return isDictionary
           ? new TypesFromInt96PageReader(dictionary, isUtcTimestamp)
           : new TypesFromInt96PageReader(valuesReader, isUtcTimestamp);
-    } else {
-      return isDictionary
-          ? new DefaultParquetDataColumnReader(dictionary)
-          : new DefaultParquetDataColumnReader(valuesReader);
     }
+    if (typeName == PrimitiveType.PrimitiveTypeName.INT64) {
+      ChronoUnit unit = resolveInt64TimestampUnit(parquetType);
+      if (unit != null) {
+        return isDictionary
+            ? new TypesFromInt64PageReader(dictionary, isUtcTimestamp, unit)
+            : new TypesFromInt64PageReader(valuesReader, isUtcTimestamp, unit);
+      }
+    }
+    return isDictionary
+        ? new DefaultParquetDataColumnReader(dictionary)
+        : new DefaultParquetDataColumnReader(valuesReader);
+  }
+
+  /**
+   * Returns the {@link ChronoUnit} for a Parquet INT64 TIMESTAMP column, or {@code null} if the
+   * column is a plain INT64 (not a timestamp).
+   *
+   * <p>Supports both the modern {@link LogicalTypeAnnotation.TimestampLogicalTypeAnnotation} and
+   * the legacy {@link OriginalType#TIMESTAMP_MILLIS} / {@link OriginalType#TIMESTAMP_MICROS}
+   * encodings.
+   */
+  private static ChronoUnit resolveInt64TimestampUnit(PrimitiveType parquetType) {
+    LogicalTypeAnnotation annotation = parquetType.getLogicalTypeAnnotation();
+    if (annotation instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+      LogicalTypeAnnotation.TimeUnit unit =
+          ((LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) annotation).getUnit();
+      switch (unit) {
+        case MILLIS:
+          return ChronoUnit.MILLIS;
+        case MICROS:
+          return ChronoUnit.MICROS;
+        case NANOS:
+          return ChronoUnit.NANOS;
+        default:
+          return null;
+      }
+    }
+    OriginalType originalType = parquetType.getOriginalType();
+    if (originalType == OriginalType.TIMESTAMP_MILLIS) {
+      return ChronoUnit.MILLIS;
+    }
+    if (originalType == OriginalType.TIMESTAMP_MICROS) {
+      return ChronoUnit.MICROS;
+    }
+    return null;
+  }
+
+  private static TimestampData int64ToTimestamp(
+      boolean utcTimestamp, long value, ChronoUnit unit) {
+    Instant instant = Instant.EPOCH.plus(value, unit);
+    if (utcTimestamp) {
+      return TimestampData.fromInstant(instant);
+    }
+    return TimestampData.fromTimestamp(Timestamp.from(instant));
   }
 
   public static ParquetDataColumnReader getDataColumnReaderByTypeOnDictionary(
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/TestHeapColumnVectorAccessors.java b/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/TestHeapColumnVectorAccessors.java
new file mode 100644
index 000000000000..4b48fdd37460
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/TestHeapColumnVectorAccessors.java
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.format.cow.vector;
+
+import org.apache.flink.table.data.columnar.vector.heap.HeapIntVector;
+import org.apache.flink.table.data.columnar.vector.heap.HeapLongVector;
+import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertSame;
+
+/**
+ * Tests for the Flink 2.1-compatible accessors added on {@link HeapArrayVector},
+ * {@link HeapMapColumnVector} and {@link HeapRowColumnVector} when vendoring Flink 2.1's
+ * nested-Parquet reader (FLINK-35702).
+ *
+ * <p>The accessors are wrappers over the existing public fields so legacy callers continue to
+ * work. These tests exist solely to pin down that wrapper contract β€” runtime correctness of the
+ * Dremel-style read path is exercised end-to-end by integration tests once
+ * {@code ParquetSplitReaderUtil} is wired up to {@code NestedColumnReader} in a follow-up PR.
+ */
+class TestHeapColumnVectorAccessors {
+
+  // -----------------------------------------------------------------------------------------------
+  // HeapArrayVector
+  // -----------------------------------------------------------------------------------------------
+
+  @Test
+  void heapArrayVectorAccessorsReflectPublicFields() {
+    HeapIntVector child = new HeapIntVector(4);
+    HeapArrayVector vector = new HeapArrayVector(2, child);
+
+    long[] offsets = {0L, 2L};
+    long[] lengths = {2L, 2L};
+    HeapLongVector replacementChild = new HeapLongVector(4);
+
+    vector.setOffsets(offsets);
+    vector.setLengths(lengths);
+    vector.setChild(replacementChild);
+    vector.setSize(2);
+
+    assertArrayEquals(offsets, vector.getOffsets());
+    assertArrayEquals(lengths, vector.getLengths());
+    assertSame(replacementChild, vector.getChild());
+    assertEquals(2, vector.getSize());
+
+    // Backing public fields are kept in sync β€” preserves backward compatibility.
+    assertSame(offsets, vector.offsets);
+    assertSame(lengths, vector.lengths);
+    assertSame(replacementChild, vector.child);
+  }
+
+  // -----------------------------------------------------------------------------------------------
+  // HeapMapColumnVector
+  // -----------------------------------------------------------------------------------------------
+
+  @Test
+  void heapMapColumnVectorConstructorInitializesOffsetsAndLengths() {
+    HeapIntVector keys = new HeapIntVector(4);
+    HeapIntVector values = new HeapIntVector(4);
+
+    HeapMapColumnVector vector = new HeapMapColumnVector(3, keys, values);
+
+    assertEquals(3, vector.getOffsets().length);
+    assertEquals(3, vector.getLengths().length);
+  }
+
+  @Test
+  void heapMapColumnVectorAccessorsReflectInternalState() {
+    HeapIntVector keys = new HeapIntVector(4);
+    HeapIntVector values = new HeapIntVector(4);
+    HeapMapColumnVector vector = new HeapMapColumnVector(2, keys, values);
+
+    long[] offsets = {0L, 2L};
+    long[] lengths = {2L, 2L};
+    HeapLongVector newKeys = new HeapLongVector(4);
+    HeapLongVector newValues = new HeapLongVector(4);
+
+    vector.setOffsets(offsets);
+    vector.setLengths(lengths);
+    vector.setKeys(newKeys);
+    vector.setValues(newValues);
+    vector.setSize(2);
+
+    assertArrayEquals(offsets, vector.getOffsets());
+    assertArrayEquals(lengths, vector.getLengths());
+    assertSame(newKeys, vector.getKeys());
+    assertSame(newValues, vector.getValues());
+    // The Flink-2.1-style ColumnVector accessors return the same underlying child.
+    assertSame(newKeys, vector.getKeyColumnVector());
+    assertSame(newValues, vector.getValueColumnVector());
+    assertEquals(2, vector.getSize());
+  }
+
+  // -----------------------------------------------------------------------------------------------
+  // HeapRowColumnVector
+  // -----------------------------------------------------------------------------------------------
+
+  @Test
+  void heapRowColumnVectorFieldsAccessorsReflectPublicVectors() {
+    HeapIntVector intField = new HeapIntVector(2);
+    HeapLongVector longField = new HeapLongVector(2);
+    HeapRowColumnVector vector = new HeapRowColumnVector(2, intField, longField);
+
+    WritableColumnVector[] originalFields = vector.getFields();
+    assertEquals(2, originalFields.length);
+    assertSame(intField, originalFields[0]);
+    assertSame(longField, originalFields[1]);
+    // Backing public field is kept in sync β€” preserves backward compatibility.
+    assertSame(originalFields, vector.vectors);
+
+    HeapIntVector replacement = new HeapIntVector(2);
+    WritableColumnVector[] replacementFields = {replacement, longField};
+    vector.setFields(replacementFields);
+
+    assertSame(replacementFields, vector.getFields());
+    assertSame(replacementFields, vector.vectors);
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/reader/TestParquetDataColumnReaderFactory.java b/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/reader/TestParquetDataColumnReaderFactory.java
new file mode 100644
index 000000000000..9d6607d03feb
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/reader/TestParquetDataColumnReaderFactory.java
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.format.cow.vector.reader;
+
+import org.apache.flink.table.data.TimestampData;
+
+import org.apache.parquet.column.Dictionary;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.OriginalType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Types;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+/**
+ * Tests for the {@link ParquetDataColumnReaderFactory} INT64 timestamp dispatch added when
+ * vendoring Flink 2.1's nested-Parquet reader (FLINK-35702).
+ *
+ * <p>The factory is exercised end-to-end by integration tests through
+ * {@link NestedPrimitiveColumnReader}; this unit test focuses on the small, deterministic piece
+ * that was added by this PR β€” selecting the right {@code ParquetDataColumnReader} for each
+ * supported INT64 TIMESTAMP encoding (modern {@link LogicalTypeAnnotation.TimestampLogicalTypeAnnotation}
+ * MILLIS / MICROS / NANOS plus the legacy {@link OriginalType} encodings) and decoding values
+ * using both the values-reader and dictionary code paths.
+ */
+class TestParquetDataColumnReaderFactory {
+
+  // -----------------------------------------------------------------------------------------------
+  // Type dispatch
+  // -----------------------------------------------------------------------------------------------
+
+  @Test
+  void valuesReaderDispatchInt96TimestampUsesInt96Reader() {
+    PrimitiveType type = Types.required(PrimitiveType.PrimitiveTypeName.INT96).named("ts");
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(type, new StubValuesReader(), true);
+    assertInstanceOf(ParquetDataColumnReaderFactory.TypesFromInt96PageReader.class, reader);
+  }
+
+  @Test
+  void valuesReaderDispatchInt64WithoutAnnotationUsesDefaultReader() {
+    PrimitiveType type = Types.required(PrimitiveType.PrimitiveTypeName.INT64).named("plainLong");
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(type, new StubValuesReader(), true);
+    assertInstanceOf(ParquetDataColumnReaderFactory.DefaultParquetDataColumnReader.class, reader);
+  }
+
+  @Test
+  void valuesReaderDispatchInt64TimestampMillisLogicalUsesInt64Reader() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(LogicalTypeAnnotation.timestampType(true, LogicalTypeAnnotation.TimeUnit.MILLIS))
+            .named("ts");
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(type, new StubValuesReader(), true);
+    assertInstanceOf(ParquetDataColumnReaderFactory.TypesFromInt64PageReader.class, reader);
+  }
+
+  @Test
+  void valuesReaderDispatchInt64TimestampMicrosLogicalUsesInt64Reader() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(LogicalTypeAnnotation.timestampType(false, LogicalTypeAnnotation.TimeUnit.MICROS))
+            .named("ts");
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(type, new StubValuesReader(), true);
+    assertInstanceOf(ParquetDataColumnReaderFactory.TypesFromInt64PageReader.class, reader);
+  }
+
+  @Test
+  void valuesReaderDispatchInt64TimestampNanosLogicalUsesInt64Reader() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(LogicalTypeAnnotation.timestampType(true, LogicalTypeAnnotation.TimeUnit.NANOS))
+            .named("ts");
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(type, new StubValuesReader(), true);
+    assertInstanceOf(ParquetDataColumnReaderFactory.TypesFromInt64PageReader.class, reader);
+  }
+
+  @Test
+  void valuesReaderDispatchInt64LegacyTimestampMillisOriginalUsesInt64Reader() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(OriginalType.TIMESTAMP_MILLIS)
+            .named("ts");
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(type, new StubValuesReader(), true);
+    assertInstanceOf(ParquetDataColumnReaderFactory.TypesFromInt64PageReader.class, reader);
+  }
+
+  @Test
+  void valuesReaderDispatchInt64LegacyTimestampMicrosOriginalUsesInt64Reader() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(OriginalType.TIMESTAMP_MICROS)
+            .named("ts");
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(type, new StubValuesReader(), true);
+    assertInstanceOf(ParquetDataColumnReaderFactory.TypesFromInt64PageReader.class, reader);
+  }
+
+  @Test
+  void valuesReaderDispatchInt32DoesNotUseTimestampReader() {
+    PrimitiveType type = Types.required(PrimitiveType.PrimitiveTypeName.INT32).named("i");
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(type, new StubValuesReader(), true);
+    assertInstanceOf(ParquetDataColumnReaderFactory.DefaultParquetDataColumnReader.class, reader);
+  }
+
+  @Test
+  void dictionaryReaderDispatchInt64TimestampMillisLogicalUsesInt64Reader() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(LogicalTypeAnnotation.timestampType(true, LogicalTypeAnnotation.TimeUnit.MILLIS))
+            .named("ts");
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByTypeOnDictionary(
+            type, new StubDictionary(), true);
+    assertInstanceOf(ParquetDataColumnReaderFactory.TypesFromInt64PageReader.class, reader);
+  }
+
+  // -----------------------------------------------------------------------------------------------
+  // INT64 β†’ TimestampData decoding (per ChronoUnit, both UTC and local-time-zone branches)
+  // -----------------------------------------------------------------------------------------------
+
+  @Test
+  void int64ReaderReadsTimestampMillisFromValuesReaderInUtc() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(LogicalTypeAnnotation.timestampType(true, LogicalTypeAnnotation.TimeUnit.MILLIS))
+            .named("ts");
+    long epochMillis = 1_700_000_000_123L;
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(
+            type, new StubValuesReader(epochMillis), true);
+
+    TimestampData ts = reader.readTimestamp();
+    assertNotNull(ts);
+    assertEquals(epochMillis, ts.getMillisecond());
+    assertEquals(0, ts.getNanoOfMillisecond());
+  }
+
+  @Test
+  void int64ReaderReadsTimestampMicrosFromValuesReaderInUtc() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(LogicalTypeAnnotation.timestampType(true, LogicalTypeAnnotation.TimeUnit.MICROS))
+            .named("ts");
+    long epochMicros = 1_700_000_000_123_456L;
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(
+            type, new StubValuesReader(epochMicros), true);
+
+    TimestampData ts = reader.readTimestamp();
+    assertNotNull(ts);
+    assertEquals(epochMicros / 1_000L, ts.getMillisecond());
+    // 456 microseconds remain β†’ 456_000 nanoseconds within the millisecond
+    assertEquals(456_000, ts.getNanoOfMillisecond());
+  }
+
+  @Test
+  void int64ReaderReadsTimestampNanosFromValuesReaderInUtc() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(LogicalTypeAnnotation.timestampType(true, LogicalTypeAnnotation.TimeUnit.NANOS))
+            .named("ts");
+    long epochNanos = 1_700_000_000_123_456_789L;
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByType(
+            type, new StubValuesReader(epochNanos), true);
+
+    TimestampData ts = reader.readTimestamp();
+    assertNotNull(ts);
+    assertEquals(epochNanos / 1_000_000L, ts.getMillisecond());
+    assertEquals(456_789, ts.getNanoOfMillisecond());
+  }
+
+  @Test
+  void int64ReaderReadsTimestampMillisFromDictionaryInUtc() {
+    PrimitiveType type =
+        Types.required(PrimitiveType.PrimitiveTypeName.INT64)
+            .as(LogicalTypeAnnotation.timestampType(true, LogicalTypeAnnotation.TimeUnit.MILLIS))
+            .named("ts");
+    long epochMillis = 1_700_000_000_456L;
+    ParquetDataColumnReader reader =
+        ParquetDataColumnReaderFactory.getDataColumnReaderByTypeOnDictionary(
+            type, new StubDictionary(epochMillis), true);
+
+    TimestampData ts = reader.readTimestamp(0);
+    assertNotNull(ts);
+    assertEquals(epochMillis, ts.getMillisecond());
+  }
+
+  // -----------------------------------------------------------------------------------------------
+  // Stubs (only the methods exercised by the dispatch + decoding tests above)
+  // -----------------------------------------------------------------------------------------------
+
+  /** Minimal {@link ValuesReader} returning a fixed long; other methods throw. */
+  private static final class StubValuesReader extends ValuesReader {
+    private final long fixedLong;
+
+    StubValuesReader() {
+      this(0L);
+    }
+
+    StubValuesReader(long fixedLong) {
+      this.fixedLong = fixedLong;
+    }
+
+    @Override
+    public long readLong() {
+      return fixedLong;
+    }
+
+    @Override
+    public void skip() {
+      // unused
+    }
+  }
+
+  /** Minimal {@link Dictionary} returning a fixed long for any id; other methods throw. */
+  private static final class StubDictionary extends Dictionary {
+    private final long fixedLong;
+
+    StubDictionary() {
+      this(0L);
+    }
+
+    StubDictionary(long fixedLong) {
+      super(null);
+      this.fixedLong = fixedLong;
+    }
+
+    @Override
+    public Binary decodeToBinary(int id) {
+      throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long decodeToLong(int id) {
+      return fixedLong;
+    }
+
+    @Override
+    public int getMaxId() {
+      return 0;
+    }
+  }
+}


(hudi) 11/16: feat: add variant type adapter for Flink (#18702)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 53559242345bead1a1dff3aac60d832605b0ce97
Author: Danny Chan <yu...@gmail.com>
AuthorDate: Mon May 11 13:44:04 2026 +0800

    feat: add variant type adapter for Flink (#18702)
    
    * Add variant type adapter for Flink
    
    * address the review comments
    
    (cherry picked from commit 5e72c96fae6117135e8b9d3a405c7d9c06003a65)
---
 .../hudi/client/model/AbstractHoodieRowData.java   |  3 +-
 .../apache/hudi/client/model/BootstrapRowData.java |  6 ++-
 .../row/parquet/ParquetSchemaConverter.java        | 47 ++++++++++++++++++
 .../apache/hudi/util/AvroToRowDataConverters.java  | 17 +++++++
 .../apache/hudi/util/HoodieSchemaConverter.java    | 24 ++++++----
 .../apache/hudi/util/RowDataToAvroConverters.java  | 28 +++++++++++
 .../hudi/util/TestHoodieSchemaConverter.java       |  3 ++
 .../ITTestVariantCrossEngineCompatibility.java     |  4 ++
 .../org/apache/hudi/adapter/DataTypeAdapter.java   | 53 ++++++++++++++++++++
 .../org/apache/hudi/adapter/DataTypeAdapter.java   | 53 ++++++++++++++++++++
 .../org/apache/hudi/adapter/DataTypeAdapter.java   | 53 ++++++++++++++++++++
 .../org/apache/hudi/adapter/DataTypeAdapter.java   | 53 ++++++++++++++++++++
 .../org/apache/hudi/adapter/DataTypeAdapter.java   | 53 ++++++++++++++++++++
 .../org/apache/hudi/adapter/DataTypeAdapter.java   | 56 ++++++++++++++++++++++
 14 files changed, 441 insertions(+), 12 deletions(-)

diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/AbstractHoodieRowData.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/AbstractHoodieRowData.java
index 94b13ea32a8a..d01bf3e34d76 100644
--- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/AbstractHoodieRowData.java
+++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/AbstractHoodieRowData.java
@@ -18,6 +18,7 @@
 
 package org.apache.hudi.client.model;
 
+import org.apache.hudi.adapter.DataTypeAdapter;
 import org.apache.hudi.common.model.HoodieOperation;
 import org.apache.hudi.common.util.ValidationUtils;
 
@@ -169,6 +170,6 @@ public abstract class AbstractHoodieRowData implements RowData {
   protected abstract int rebaseOrdinal(int ordinal);
 
   public Variant getVariant(int i) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+    return DataTypeAdapter.getVariant(row, rebaseOrdinal(i));
   }
 }
diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/BootstrapRowData.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/BootstrapRowData.java
index 92c0f35a4232..23c4c82f9668 100644
--- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/BootstrapRowData.java
+++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/model/BootstrapRowData.java
@@ -18,6 +18,8 @@
 
 package org.apache.hudi.client.model;
 
+import org.apache.hudi.adapter.DataTypeAdapter;
+
 import org.apache.flink.table.data.ArrayData;
 import org.apache.flink.table.data.DecimalData;
 import org.apache.flink.table.data.MapData;
@@ -149,7 +151,7 @@ public class BootstrapRowData implements RowData {
     return getter.apply(pos);
   }
 
-  public Variant getVariant(int i) {
-    throw new UnsupportedOperationException("Variant is not supported yet.");
+  public Variant getVariant(int pos) {
+    return DataTypeAdapter.getVariant(row, pos);
   }
 }
diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetSchemaConverter.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetSchemaConverter.java
index d98497fdc86c..668098314d48 100644
--- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetSchemaConverter.java
+++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/parquet/ParquetSchemaConverter.java
@@ -18,6 +18,8 @@
 
 package org.apache.hudi.io.storage.row.parquet;
 
+import org.apache.hudi.adapter.DataTypeAdapter;
+import org.apache.hudi.common.schema.HoodieSchema;
 import org.apache.hudi.common.util.collection.Pair;
 
 import lombok.extern.slf4j.Slf4j;
@@ -155,6 +157,15 @@ public class ParquetSchemaConverter {
             new MapType(
                 convertToRowField(keyValueType.getLeft()).getType().copy(true),
                 convertToRowField(keyValueType.getRight()).getType()));
+      } else if (hasVariantAnnotation(logicalType)) {
+        if (isShreddedVariant(groupType)) {
+          throw new UnsupportedOperationException(
+              "Shredded Variant is not supported in Flink. "
+                  + "The Parquet group '" + groupType.getName() + "' contains a '"
+                  + HoodieSchema.Variant.VARIANT_TYPED_VALUE_FIELD
+                  + "' field indicating a shredded layout.");
+        }
+        dataType = DataTypeAdapter.createVariantType();
       } else {
         dataType =
             DataTypes.of(new RowType(
@@ -190,6 +201,39 @@ public class ParquetSchemaConverter {
     return new MessageType(name, types);
   }
 
+  /**
+   * Checks whether the group carries the Parquet {@code VARIANT} logical type annotation.
+   * Uses class-name matching so this compiles against parquet-java versions that predate the
+   * {@code VariantLogicalTypeAnnotation} class (< 1.15.2).
+   */
+  private static boolean hasVariantAnnotation(LogicalTypeAnnotation logicalType) {
+    // needs to ensure the writer attach the variant annotation in 1.3.
+    return logicalType != null
+        && logicalType.getClass().getSimpleName().equals("VariantLogicalTypeAnnotation");
+  }
+
+  /**
+   * Checks whether a variant group contains a {@code typed_value} field, indicating a shredded
+   * layout. Called only after {@link #hasVariantAnnotation} returns true.
+   */
+  private static boolean isShreddedVariant(GroupType groupType) {
+    return groupType.containsField(HoodieSchema.Variant.VARIANT_TYPED_VALUE_FIELD);
+  }
+
+  /**
+   * Converts a Variant column to the canonical unshredded Parquet layout:
+   * a group with required binary {@code metadata} and required binary {@code value}.
+   */
+  private static Type convertVariantToParquetType(String name, Type.Repetition repetition) {
+    // TODO: add .as(LogicalTypeAnnotation.variantType()) once parquet-java is bumped to 1.16.0
+    return Types.buildGroup(repetition)
+        .addField(Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY, Type.Repetition.REQUIRED)
+            .named(HoodieSchema.Variant.VARIANT_METADATA_FIELD))
+        .addField(Types.primitive(PrimitiveType.PrimitiveTypeName.BINARY, Type.Repetition.REQUIRED)
+            .named(HoodieSchema.Variant.VARIANT_VALUE_FIELD))
+        .named(name);
+  }
+
   private static Type convertToParquetType(
       String name, LogicalType type, Type.Repetition repetition) {
     switch (type.getTypeRoot()) {
@@ -304,6 +348,9 @@ public class ParquetSchemaConverter {
             .addField(convertToParquetType(field.getName(), field.getType(), field.getType().isNullable() ? Type.Repetition.OPTIONAL : Type.Repetition.REQUIRED)));
         return builder.named(name);
       default:
+        if (DataTypeAdapter.isVariantType(type)) {
+          return convertVariantToParquetType(name, repetition);
+        }
         throw new UnsupportedOperationException("Unsupported type: " + type);
     }
   }
diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/AvroToRowDataConverters.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/AvroToRowDataConverters.java
index fee24e520382..d63ce350b487 100644
--- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/AvroToRowDataConverters.java
+++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/AvroToRowDataConverters.java
@@ -18,6 +18,8 @@
 
 package org.apache.hudi.util;
 
+import org.apache.hudi.adapter.DataTypeAdapter;
+
 import lombok.AccessLevel;
 import lombok.NoArgsConstructor;
 import org.apache.avro.generic.GenericFixed;
@@ -155,6 +157,9 @@ public class AvroToRowDataConverters {
       case MULTISET:
         return createMapConverter(type, utcTimezone);
       default:
+        if (DataTypeAdapter.isVariantType(type)) {
+          return createVariantConverter();
+        }
         throw new UnsupportedOperationException("Unsupported type: " + type);
     }
   }
@@ -212,6 +217,18 @@ public class AvroToRowDataConverters {
     };
   }
 
+  /**
+   * Creates a converter for Flink 2.1+ VARIANT LogicalType. The converter receives an Avro
+   * GenericRecord carrying metadata/value binary fields and produces a Flink
+   * {@code BinaryVariant}.
+   */
+  private static AvroToRowDataConverter createVariantConverter() {
+    return avroObject -> {
+      IndexedRecord record = (IndexedRecord) avroObject;
+      return DataTypeAdapter.createVariant(convertToBytes(record.get(1)), convertToBytes(record.get(0)));
+    };
+  }
+
   private static AvroToRowDataConverter createTimestampConverter(int precision, boolean utcTimezone) {
     final ChronoUnit chronoUnit;
     if (precision <= 3) {
diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/HoodieSchemaConverter.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/HoodieSchemaConverter.java
index 0b3a1af7982b..c49142800b45 100644
--- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/HoodieSchemaConverter.java
+++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/HoodieSchemaConverter.java
@@ -19,6 +19,7 @@
 
 package org.apache.hudi.util;
 
+import org.apache.hudi.adapter.DataTypeAdapter;
 import org.apache.hudi.common.schema.HoodieSchema;
 import org.apache.hudi.common.schema.HoodieSchemaField;
 import org.apache.hudi.common.schema.HoodieSchemaType;
@@ -222,6 +223,10 @@ public class HoodieSchemaConverter {
 
       case RAW:
       default:
+        if (DataTypeAdapter.isVariantType(logicalType)) {
+          schema = HoodieSchema.createVariant();
+          break;
+        }
         throw new UnsupportedOperationException(
             "Unsupported type for HoodieSchema conversion: " + logicalType);
     }
@@ -576,23 +581,24 @@ public class HoodieSchemaConverter {
   }
 
   /**
-   * Converts a Variant schema to Flink's ROW type.
-   * Variant is represented as ROW<`metadata` BYTES, `value` BYTES> in Flink.
+   * Converts a Variant HoodieSchema to the native Flink {@code VariantType} DataType.
+   * Requires Flink 2.1+ at runtime; throws {@link UnsupportedOperationException} on older versions.
    *
    * @param schema HoodieSchema to convert (must be a VARIANT type)
-   * @return DataType representing the Variant as a ROW with binary fields
+   * @return native VariantType DataType
+   * @throws UnsupportedOperationException if Flink runtime is pre-2.1 or variant is shredded
    */
   private static DataType convertVariant(HoodieSchema schema) {
     if (schema.getType() != HoodieSchemaType.VARIANT) {
       throw new IllegalStateException("Expected HoodieSchema.Variant but got: " + schema.getClass());
     }
 
-    // Variant is stored as a struct with two binary fields: metadata and value.
-    // Field order follows the Parquet spec and Iceberg convention (metadata first, value second).
-    return DataTypes.ROW(
-        DataTypes.FIELD("metadata", DataTypes.BYTES().notNull()),
-        DataTypes.FIELD("value", DataTypes.BYTES().notNull())
-    ).notNull();
+    if (((HoodieSchema.Variant) schema).isShredded()) {
+      throw new UnsupportedOperationException(
+          "Shredded Variant is not yet supported in Flink. Use unshredded Variant instead.");
+    }
+
+    return DataTypeAdapter.createVariantType().notNull();
   }
 
   /**
diff --git a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/RowDataToAvroConverters.java b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/RowDataToAvroConverters.java
index 9eb2979e19aa..6bf94b527074 100644
--- a/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/RowDataToAvroConverters.java
+++ b/hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/util/RowDataToAvroConverters.java
@@ -18,6 +18,7 @@
 
 package org.apache.hudi.util;
 
+import org.apache.hudi.adapter.DataTypeAdapter;
 import org.apache.hudi.common.schema.HoodieSchema;
 import org.apache.hudi.common.schema.HoodieSchemaField;
 import org.apache.hudi.common.schema.HoodieSchemaType;
@@ -239,6 +240,10 @@ public class RowDataToAvroConverters {
         break;
       case RAW:
       default:
+        if (DataTypeAdapter.isVariantType(type)) {
+          converter = createVariantConverter();
+          break;
+        }
         throw new UnsupportedOperationException("Unsupported type: " + type);
     }
 
@@ -357,5 +362,28 @@ public class RowDataToAvroConverters {
       }
     };
   }
+
+  /**
+   * Creates a converter for Flink 2.1+ VARIANT LogicalType. The converter receives a Flink
+   * {@code Variant} object at runtime and extracts the raw metadata/value byte arrays,
+   * then packs them into an Avro GenericRecord with the Variant schema.
+   *
+   * <p>No shredded-variant check is needed here: {@code HoodieSchemaConverter.convertVariant()}
+   * already rejects shredded variants before a Flink type or converter is ever constructed,
+   * and Flink 2.1 itself only supports unshredded variants (FLIP-521).
+   */
+  private static RowDataToAvroConverter createVariantConverter() {
+    return new RowDataToAvroConverter() {
+      private static final long serialVersionUID = 1L;
+
+      @Override
+      public Object convert(HoodieSchema schema, Object object) {
+        final GenericRecord record = new GenericData.Record(schema.toAvroSchema());
+        record.put(HoodieSchema.Variant.VARIANT_METADATA_FIELD, ByteBuffer.wrap(DataTypeAdapter.getVariantMetadata(object)));
+        record.put(HoodieSchema.Variant.VARIANT_VALUE_FIELD, ByteBuffer.wrap(DataTypeAdapter.getVariantValue(object)));
+        return record;
+      }
+    };
+  }
 }
 
diff --git a/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java b/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java
index e2fd16d7112b..51b8a15f4c7b 100644
--- a/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java
+++ b/hudi-client/hudi-flink-client/src/test/java/org/apache/hudi/util/TestHoodieSchemaConverter.java
@@ -36,6 +36,7 @@ import org.apache.flink.table.types.logical.RowType;
 import org.apache.flink.table.types.logical.TimestampType;
 import org.apache.flink.table.types.logical.VarCharType;
 import org.apache.flink.table.types.logical.VarBinaryType;
+import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 
 import java.util.Arrays;
@@ -692,6 +693,7 @@ public class TestHoodieSchemaConverter {
   }
 
   @Test
+  @Disabled("disabled and reopen the tests for 1.3")
   public void testVariantTypeConversion() {
     // Test direct Variant conversion
     HoodieSchema variantSchema = HoodieSchema.createVariant();
@@ -708,6 +710,7 @@ public class TestHoodieSchemaConverter {
   }
 
   @Test
+  @Disabled("disabled and reopen the tests for 1.3")
   public void testVariantInRecordConversion() {
     // Test Variant field within a record
     HoodieSchema recordWithVariant = HoodieSchema.createRecord(
diff --git a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java
index 9aade5503b4c..03305d478ecd 100644
--- a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java
+++ b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestVariantCrossEngineCompatibility.java
@@ -27,6 +27,7 @@ import org.apache.flink.table.api.TableEnvironment;
 import org.apache.flink.table.api.TableResult;
 import org.apache.flink.types.Row;
 import org.apache.flink.util.CollectionUtil;
+import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.extension.ExtendWith;
 import org.junit.jupiter.api.io.TempDir;
@@ -114,6 +115,7 @@ public class ITTestVariantCrossEngineCompatibility {
   }
 
   @Test
+  @Disabled("disabled and reopen the tests for 1.3")
   public void testFlinkReadSparkVariantCOWTable() throws Exception {
     // Test that Flink can read a COW table with Variant data written by Spark 4.0
     Path cowTargetDir = tempDir.resolve("cow");
@@ -123,6 +125,7 @@ public class ITTestVariantCrossEngineCompatibility {
   }
 
   @Test
+  @Disabled("disabled and reopen the tests for 1.3")
   public void testFlinkReadSparkVariantMORTableWithAvro() throws Exception {
     // Test that Flink can read a MOR table with AVRO record type and Variant data written by Spark 4.0
     Path morAvroTargetDir = tempDir.resolve("mor_avro");
@@ -132,6 +135,7 @@ public class ITTestVariantCrossEngineCompatibility {
   }
 
   @Test
+  @Disabled("disabled and reopen the tests for 1.3")
   public void testFlinkReadSparkVariantMORTableWithSpark() throws Exception {
     // Test that Flink can read a MOR table with SPARK record type and Variant data written by Spark 4.0
     Path morSparkTargetDir = tempDir.resolve("mor_spark");
diff --git a/hudi-flink-datasource/hudi-flink1.17.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink1.17.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
new file mode 100644
index 000000000000..79b9e254dd61
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.17.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.types.variant.Variant;
+
+/**
+ * Adapter utils to provide {@code DataType} utilities.
+ */
+public class DataTypeAdapter {
+  public static Variant getVariant(RowData rowData, int pos) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static Object createVariant(byte[] value, byte[] metadata) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static boolean isVariantType(LogicalType logicalType) {
+    return false;
+  }
+
+  public static DataType createVariantType() {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantMetadata(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantValue(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
new file mode 100644
index 000000000000..79b9e254dd61
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.types.variant.Variant;
+
+/**
+ * Adapter utils to provide {@code DataType} utilities.
+ */
+public class DataTypeAdapter {
+  public static Variant getVariant(RowData rowData, int pos) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static Object createVariant(byte[] value, byte[] metadata) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static boolean isVariantType(LogicalType logicalType) {
+    return false;
+  }
+
+  public static DataType createVariantType() {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantMetadata(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantValue(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink1.19.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink1.19.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
new file mode 100644
index 000000000000..79b9e254dd61
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.19.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.types.variant.Variant;
+
+/**
+ * Adapter utils to provide {@code DataType} utilities.
+ */
+public class DataTypeAdapter {
+  public static Variant getVariant(RowData rowData, int pos) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static Object createVariant(byte[] value, byte[] metadata) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static boolean isVariantType(LogicalType logicalType) {
+    return false;
+  }
+
+  public static DataType createVariantType() {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantMetadata(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantValue(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink1.20.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink1.20.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
new file mode 100644
index 000000000000..79b9e254dd61
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink1.20.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.types.variant.Variant;
+
+/**
+ * Adapter utils to provide {@code DataType} utilities.
+ */
+public class DataTypeAdapter {
+  public static Variant getVariant(RowData rowData, int pos) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static Object createVariant(byte[] value, byte[] metadata) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static boolean isVariantType(LogicalType logicalType) {
+    return false;
+  }
+
+  public static DataType createVariantType() {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantMetadata(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantValue(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
diff --git a/hudi-flink-datasource/hudi-flink2.0.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink2.0.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
new file mode 100644
index 000000000000..566ca723648a
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink2.0.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.types.variant.Variant;
+
+/**
+ * Adapter utils to provide {@code DataType} utilities.
+ */
+public class DataTypeAdapter {
+  public static Variant getVariant(RowData rowData, int pos) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static Object createVariant(byte[] value, byte[] metadata) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static boolean isVariantType(LogicalType logicalType) {
+    return false;
+  }
+
+  public static DataType createVariantType() {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantMetadata(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+
+  public static byte[] getVariantValue(Object obj) {
+    throw new UnsupportedOperationException("Variant is not supported yet.");
+  }
+}
\ No newline at end of file
diff --git a/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java b/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
new file mode 100644
index 000000000000..6ae4c4a6baba
--- /dev/null
+++ b/hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/adapter/DataTypeAdapter.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.adapter;
+
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.LogicalTypeRoot;
+import org.apache.flink.types.variant.BinaryVariant;
+import org.apache.flink.types.variant.Variant;
+
+/**
+ * Adapter utils to provide {@code DataType} utilities.
+ */
+public class DataTypeAdapter {
+  public static Variant getVariant(RowData rowData, int pos) {
+    return rowData.getVariant(pos);
+  }
+
+  public static Object createVariant(byte[] value, byte[] metadata) {
+    return new BinaryVariant(value, metadata);
+  }
+
+  public static boolean isVariantType(LogicalType logicalType) {
+    return logicalType.getTypeRoot() == LogicalTypeRoot.VARIANT;
+  }
+
+  public static DataType createVariantType() {
+    return DataTypes.VARIANT();
+  }
+
+  public static byte[] getVariantMetadata(Object obj) {
+    return ((BinaryVariant) obj).getMetadata();
+  }
+
+  public static byte[] getVariantValue(Object obj) {
+    return ((BinaryVariant) obj).getValue();
+  }
+}


(hudi) 14/16: [MINOR] Fix typos in comments and assertion messages (#18763)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 9023d94478d5c68cf8269e08fc07bc3f9a22b425
Author: Xinli Shang <sh...@uber.com>
AuthorDate: Sun May 17 20:15:40 2026 -0700

    [MINOR] Fix typos in comments and assertion messages (#18763)
    
    Corrects a handful of long-standing typos in code comments and
    test assertion messages across the codebase. No functional changes.
    
    - atleast -> at least (8 occurrences)
    - commited -> committed (1 occurrence in comment)
    - existance -> existence (1 occurrence in exception message)
    - transfering -> transferring (2 occurrences)
    - succesfully -> successfully (1 occurrence in comment)
    
    Co-authored-by: Xinli Shang <sh...@apache.org>
    (cherry picked from commit cd2c8b831c8b7c8f46b1f175bdb323329f60c8ab)
---
 .../main/java/org/apache/hudi/client/utils/LazyIterableIterator.java  | 2 +-
 .../org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java     | 2 +-
 .../test/java/org/apache/hudi/client/TestCoalescingPartitioner.java   | 2 +-
 .../src/test/java/org/apache/hudi/table/TestHoodieSparkTable.java     | 2 +-
 .../java/org/apache/hudi/table/action/compact/CompactionTestBase.java | 4 ++--
 .../testsuite/generator/GenericRecordPartialPayloadGenerator.java     | 2 +-
 .../org/apache/hudi/integ/testsuite/TestFileDeltaInputWriter.java     | 2 +-
 .../datasources/parquet/Spark41LegacyHoodieParquetFileFormat.scala    | 2 +-
 .../sql/execution/datasources/parquet/Spark41ParquetReader.scala      | 2 +-
 .../deltastreamer/TestHoodieDeltaStreamerWithMultiWriter.java         | 2 +-
 scripts/pr_compliance.py                                              | 2 +-
 11 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LazyIterableIterator.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LazyIterableIterator.java
index b921c6ddfc81..64a92ee1ae8c 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LazyIterableIterator.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/LazyIterableIterator.java
@@ -28,7 +28,7 @@ import java.util.Iterator;
  * Provide a way to obtain a inputItr of type O (output), out of an inputItr of type I (input)
  * <p>
  * Things to remember: - Assumes Spark calls hasNext() to check for elements, before calling next() to obtain them -
- * Assumes hasNext() gets called atleast once. - Concrete Implementation is responsible for calling inputIterator.next()
+ * Assumes hasNext() gets called at least once. - Concrete Implementation is responsible for calling inputIterator.next()
  * and doing the processing in computeNext()
  */
 public abstract class LazyIterableIterator<I, O> implements Iterable<O>, Iterator<O> {
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
index db30f301d97a..5d09762909be 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
@@ -2026,7 +2026,7 @@ public abstract class HoodieBackedTableMetadataWriter<I, O> implements HoodieTab
         // scheduling of INDEX only initializes the file group and not add commit
         // so if there are no committed file slices, look for inflight slices
         if (isNonGlobalRLI) {
-          // For isNonGlobalRLI, new partitions added to the data table will cause new filegroups that are not yet commited
+          // For isNonGlobalRLI, new partitions added to the data table will cause new filegroups that are not yet committed
           // therefore, we always need to look for inflight filegroups
           fileSlices = getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient, Option.ofNullable(fsView), partitionPath);
         } else if (fileSlices.isEmpty()) {
diff --git a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestCoalescingPartitioner.java b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestCoalescingPartitioner.java
index 0f3b972574ce..9f99742e0c0e 100644
--- a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestCoalescingPartitioner.java
+++ b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestCoalescingPartitioner.java
@@ -123,7 +123,7 @@ public class TestCoalescingPartitioner extends HoodieClientTestBase {
     }, true).collect();
 
     assertEquals(targetPartitions, countsPerPartition.size());
-    // lets validate that atleast we have 50% of data in each spark partition compared to ideal scenario (we can't assume hash of strings will evenly distribute).
+    // lets validate that at least we have 50% of data in each spark partition compared to ideal scenario (we can't assume hash of strings will evenly distribute).
     countsPerPartition.forEach(pair -> {
       int numElements = pair.getValue();
       int idealExpectedCount = inputNumPartitions / targetPartitions;
diff --git a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/TestHoodieSparkTable.java b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/TestHoodieSparkTable.java
index e50bb9e5ecb7..9808de708374 100644
--- a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/TestHoodieSparkTable.java
+++ b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/TestHoodieSparkTable.java
@@ -107,7 +107,7 @@ public class TestHoodieSparkTable extends HoodieCommonTestHarness {
         // lets create the data file. so that we can validate later.
         localStorage.create(storagePath);
       } catch (IOException e) {
-        throw new HoodieException("Failed to check data file existance " + fileName);
+        throw new HoodieException("Failed to check data file existence " + fileName);
       }
     });
     HoodieTable hoodieTable = HoodieSparkTable.create(writeConfig, getEngineContext(), metaClient);
diff --git a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/CompactionTestBase.java b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/CompactionTestBase.java
index ae6b60ccf679..677ece46d801 100644
--- a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/CompactionTestBase.java
+++ b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/action/compact/CompactionTestBase.java
@@ -104,7 +104,7 @@ public class CompactionTestBase extends HoodieClientTestBase {
       if (opPair != null) {
         assertEquals(fileSlice.getBaseInstantTime(), opPair.getKey(), "Expect baseInstant to match compaction Instant");
         assertTrue(fileSlice.getLogFiles().count() > 0,
-            "Expect atleast one log file to be present where the latest delta commit was written");
+            "Expect at least one log file to be present where the latest delta commit was written");
         assertFalse(fileSlice.getBaseFile().isPresent(), "Expect no data-file to be present");
       } else {
         assertTrue(fileSlice.getBaseInstantTime().compareTo(latestDeltaCommit) <= 0,
@@ -210,7 +210,7 @@ public class CompactionTestBase extends HoodieClientTestBase {
 
     if (hasDeltaCommitAfterPendingCompaction) {
       assertFalse(fileSliceList.stream().anyMatch(fs -> fs.getLogFiles().count() == 0),
-          "Verify all file-slices have atleast one log-file");
+          "Verify all file-slices have at least one log-file");
     } else {
       assertFalse(fileSliceList.stream().anyMatch(fs -> fs.getLogFiles().count() > 0),
           "Verify all file-slices have no log-files");
diff --git a/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordPartialPayloadGenerator.java b/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordPartialPayloadGenerator.java
index 999f83de66c0..b42ad4396c1b 100644
--- a/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordPartialPayloadGenerator.java
+++ b/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/generator/GenericRecordPartialPayloadGenerator.java
@@ -63,7 +63,7 @@ public class GenericRecordPartialPayloadGenerator extends GenericRecordFullPaylo
     return validate((Object) record);
   }
 
-  // Atleast 1 entry should be null
+  // At least 1 entry should be null
   private boolean validate(Object object) {
     if (object == null) {
       return true;
diff --git a/hudi-integ-test/src/test/java/org/apache/hudi/integ/testsuite/TestFileDeltaInputWriter.java b/hudi-integ-test/src/test/java/org/apache/hudi/integ/testsuite/TestFileDeltaInputWriter.java
index a33ee7c5fb4f..ad6ae1767383 100644
--- a/hudi-integ-test/src/test/java/org/apache/hudi/integ/testsuite/TestFileDeltaInputWriter.java
+++ b/hudi-integ-test/src/test/java/org/apache/hudi/integ/testsuite/TestFileDeltaInputWriter.java
@@ -98,7 +98,7 @@ public class TestFileDeltaInputWriter extends UtilitiesTestBase {
     DeltaWriteStats deltaWriteStats = fileSinkWriter.getDeltaWriteStats();
     FileSystem fs = HadoopFSUtils.getFs(basePath, jsc.hadoopConfiguration());
     FileStatus[] fileStatuses = fs.listStatus(new Path(deltaWriteStats.getFilePath()));
-    // Atleast 1 file was written
+    // At least 1 file was written
     assertEquals(1, fileStatuses.length);
     // File length should be greater than 0
     assertTrue(fileStatuses[0].getLen() > 0);
diff --git a/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark41LegacyHoodieParquetFileFormat.scala b/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark41LegacyHoodieParquetFileFormat.scala
index 8dff79c1e07b..91b085b693aa 100644
--- a/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark41LegacyHoodieParquetFileFormat.scala
+++ b/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark41LegacyHoodieParquetFileFormat.scala
@@ -187,7 +187,7 @@ class Spark41LegacyHoodieParquetFileFormat(private val shouldAppendPartitionValu
       }
 
       // When there are vectorized reads, we can avoid
-      // 1. opening the file twice by transfering the SeekableInputStream
+      // 1. opening the file twice by transferring the SeekableInputStream
       // 2. reading the footer twice by reading all row groups in advance and filter row groups
       //    according to filters that require push down
       val openedFooter = ParquetFooterReader.openFileAndReadFooter(sharedConf, file, enableVectorizedReader)
diff --git a/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark41ParquetReader.scala b/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark41ParquetReader.scala
index f5ccd7b54176..d15520106e23 100644
--- a/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark41ParquetReader.scala
+++ b/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark41ParquetReader.scala
@@ -108,7 +108,7 @@ class Spark41ParquetReader(enableVectorizedReader: Boolean,
       partitionSchema, internalSchemaOpt)
 
     // When there are vectorized reads, we can avoid
-    // 1. opening the file twice by transfering the SeekableInputStream
+    // 1. opening the file twice by transferring the SeekableInputStream
     // 2. reading the footer twice by reading all row groups in advance and filter row groups
     //    according to filters that require push down
     val originalFooter =
diff --git a/hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamerWithMultiWriter.java b/hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamerWithMultiWriter.java
index fbd8726242a6..5e09ac923794 100644
--- a/hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamerWithMultiWriter.java
+++ b/hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamerWithMultiWriter.java
@@ -419,7 +419,7 @@ public class TestHoodieDeltaStreamerWithMultiWriter extends HoodieDeltaStreamerT
       });
       Future backfillJobFuture = service.submit(() -> {
         try {
-          // trigger backfill atleast after 1 requested entry is added to timeline from continuous job. If not, there is a chance that backfill will complete even before
+          // trigger backfill at least after 1 requested entry is added to timeline from continuous job. If not, there is a chance that backfill will complete even before
           // continuous job starts.
           awaitCondition(new GetCommitsAfterInstant(tableBasePath, lastSuccessfulCommit));
           backfillJob.sync();
diff --git a/scripts/pr_compliance.py b/scripts/pr_compliance.py
index aafeb9c2a0d0..5bf1a23b69e1 100644
--- a/scripts/pr_compliance.py
+++ b/scripts/pr_compliance.py
@@ -46,7 +46,7 @@ class Outcomes:
     #parsing the next section
     NEXTSECTION = 2
 
-    #parsing has concluded succesfully, exit with no error
+    #parsing has concluded successfully, exit with no error
     SUCCESS = 3
 
 


(hudi) 03/16: feat(common): roll over commit metadata to clean (#18590)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 69ec50cc07c48aba3af7bc72233881bd82cd8702
Author: Krishen <22...@users.noreply.github.com>
AuthorDate: Mon May 4 13:33:22 2026 -0700

    feat(common): roll over commit metadata to clean (#18590)
    
    When rolling metadata is configured (hoodie.write.rolling.metadata.keys), important metadata like schema and checkpoint keys are carried forward across commits. However, clean instants do not participate in this rolling mechanism, they neither receive rolled-over metadata nor serve as a source for subsequent lookups. After archival removes old ingestion commits, if only clean instants remain on the active timeline between surviving commits, the chain of rolled-over metadata can break.
    
    This PR ensures that clean commits also carry rolled-over metadata in their extraMetadata field, preserving the rolling metadata chain across archival.
    
    ---------
    
    Co-authored-by: Krishen Bhan <β€œbkrishen@uber.com”>
    (cherry picked from commit 127c6ee031867a7a6b49c6f3ccadbba8161ee8ef)
---
 .../org/apache/hudi/client/BaseHoodieClient.java   | 170 +++++++++++++--------
 .../org/apache/hudi/config/HoodieWriteConfig.java  |  19 +--
 .../table/action/clean/CleanActionExecutor.java    |   6 +-
 .../hudi/common/table/timeline/HoodieTimeline.java |   5 +
 .../TestHoodieClientOnCopyOnWriteStorage.java      |  86 +++++++++++
 5 files changed, 210 insertions(+), 76 deletions(-)

diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieClient.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieClient.java
index 842dc38177a5..002ac1e7454f 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieClient.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieClient.java
@@ -18,6 +18,7 @@
 
 package org.apache.hudi.client;
 
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
 import org.apache.hudi.callback.HoodieClientInitCallback;
 import org.apache.hudi.client.embedded.EmbeddedTimelineServerHelper;
 import org.apache.hudi.client.embedded.EmbeddedTimelineService;
@@ -58,6 +59,7 @@ import lombok.extern.slf4j.Slf4j;
 import java.io.IOException;
 import java.io.Serializable;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
@@ -313,19 +315,24 @@ public abstract class BaseHoodieClient implements Serializable, AutoCloseable {
   }
 
   /**
-   * Merges rolling metadata from recent completed commits into the current commit metadata.
+   * Merges rolling metadata from recent completed instants into the current commit metadata.
    * This method MUST be called within the transaction lock after conflict resolution.
    *
    * <p>Rolling metadata keys configured via {@link HoodieWriteConfig#ROLLING_METADATA_KEYS} will be
-   * automatically carried forward from recent commits. The system walks back up to
-   * {@link HoodieWriteConfig#ROLLING_METADATA_TIMELINE_LOOKBACK_COMMITS} commits to find the most
-   * recent value for each key. This ensures that important metadata like checkpoint information
-   * remains accessible without worrying about archival or missing keys in individual commits.
+   * automatically carried forward from recent instants. The system walks back through completed
+   * commits and clean instants (in reverse completion-time order) up to
+   * {@link HoodieWriteConfig#ROLLING_METADATA_TIMELINE_LOOKBACK_COMMITS} to find the most
+   * recent value for each key.
    *
    * @param table HoodieTable instance (may have refreshed timeline after conflict resolution)
    * @param metadata Current commit metadata to be augmented with rolling metadata
    */
   protected void mergeRollingMetadata(HoodieTable table, HoodieCommitMetadata metadata) {
+    // IMPORTANT: We're inside the lock here. The timeline in 'table' is either:
+    // 1. Fresh from createTable() if no conflict resolution happened
+    // 2. Reloaded during resolveWriteConflict() if conflicts were checked
+    // In both cases, we have the latest view of the timeline.
+
     // Skip for metadata table - rolling metadata is only for data tables
     if (table.isMetadataTable()) {
       return;
@@ -336,88 +343,119 @@ public abstract class BaseHoodieClient implements Serializable, AutoCloseable {
       return;  // No rolling metadata configured
     }
 
-    // IMPORTANT: We're inside the lock here. The timeline in 'table' is either:
-    // 1. Fresh from createTable() if no conflict resolution happened
-    // 2. Reloaded during resolveWriteConflict() if conflicts were checked
-    // In both cases, we have the latest view of the timeline.
+    Map<String, String> foundRollingMetadata = collectRollingMetadataFromTimeline(table, config, rollingKeys, metadata.getExtraMetadata());
+    for (Map.Entry<String, String> entry : foundRollingMetadata.entrySet()) {
+      metadata.addMetadata(entry.getKey(), entry.getValue());
+    }
+  }
 
-    HoodieTimeline commitsTimeline = table.getActiveTimeline().getCommitsTimeline().filterCompletedInstants();
+  /**
+   * Overload of {@link #mergeRollingMetadata(HoodieTable, HoodieCommitMetadata)} for clean
+   * commits. Populates {@link HoodieCleanMetadata#getExtraMetadata()} with rolling metadata
+   * values found on the active timeline.
+   *
+   * <p>This is {@code public static} so that {@code CleanActionExecutor} (which does not extend
+   * {@code BaseHoodieClient}) can invoke it.
+   */
+  public static void mergeRollingMetadata(HoodieTable table, HoodieWriteConfig config, HoodieCleanMetadata metadata) {
+    if (table.isMetadataTable()) {
+      return;
+    }
+    Set<String> rollingKeys = config.getRollingMetadataKeys();
+    if (rollingKeys.isEmpty()) {
+      return;
+    }
 
-    if (commitsTimeline.empty()) {
-      log.info("No previous commits found. Rolling metadata will start with current commit.");
-      return;  // First commit - nothing to roll forward
+    Map<String, String> existing = metadata.getExtraMetadata() != null
+        ? metadata.getExtraMetadata() : Collections.emptyMap();
+    Map<String, String> foundRollingMetadata = collectRollingMetadataFromTimeline(table, config, rollingKeys, existing);
+    if (!foundRollingMetadata.isEmpty()) {
+      Map<String, String> merged = new HashMap<>(existing);
+      merged.putAll(foundRollingMetadata);
+      metadata.setExtraMetadata(merged);
     }
+  }
 
-    try {
-      Map<String, String> existingExtraMetadata = metadata.getExtraMetadata();
-      Map<String, String> foundRollingMetadata = new HashMap<>();
-      Set<String> remainingKeys = new HashSet<>(rollingKeys);
-
-      // Remove keys that are already present with non-empty values in current commit (current values take precedence)
-      for (String key : rollingKeys) {
-        if (existingExtraMetadata.containsKey(key) && !StringUtils.isNullOrEmpty(existingExtraMetadata.get(key))) {
-          remainingKeys.remove(key);
-        }
-      }
+  /**
+   * Walks backwards through completed instants (commits, replace-commits, delta-commits, and
+   * clean) on the active timeline, extracting extra-metadata values for the requested rolling
+   * keys. For commit-type instants the values come from {@link HoodieCommitMetadata#getMetadata};
+   * for clean instants they come from {@link HoodieCleanMetadata#getExtraMetadata()}.
+   *
+   * <p>Keys already present with a non-empty value in {@code existingExtra} are skipped (empty
+   * strings are treated as "missing").
+   */
+  private static Map<String, String> collectRollingMetadataFromTimeline(
+      HoodieTable table, HoodieWriteConfig config,
+      Set<String> rollingKeys, Map<String, String> existingExtra) {
 
-      if (remainingKeys.isEmpty()) {
-        log.debug("All rolling metadata keys are present in current commit. No walkback needed.");
-        return;
-      }
+    Map<String, String> foundRollingMetadata = new HashMap<>();
+    Set<String> remaining = new HashSet<>(rollingKeys);
 
-      int lookbackLimit = config.getRollingMetadataTimelineLookbackCommits();
-      int commitsWalkedBack = 0;
+    for (String key : rollingKeys) {
+      if (existingExtra.containsKey(key) && !StringUtils.isNullOrEmpty(existingExtra.get(key))) {
+        remaining.remove(key);
+      }
+    }
+    if (remaining.isEmpty()) {
+      log.debug("All rolling metadata keys already present. No walkback needed.");
+      return foundRollingMetadata;
+    }
 
-      // Walk back through the timeline in reverse order (most recent first) to find values for all remaining keys
-      List<HoodieInstant> recentCommits = commitsTimeline.getReverseOrderedInstantsByCompletionTime()
-          .limit(lookbackLimit)
-          .collect(Collectors.toList());
+    int lookbackLimit = config.getRollingMetadataTimelineLookbackCommits();
+    HoodieTimeline completed = table.getActiveTimeline().filterCompletedInstants();
+    List<HoodieInstant> instants = completed.getReverseOrderedInstantsByCompletionTime()
+        .filter(i -> HoodieTimeline.VALID_ACTIONS_FOR_ROLLING_METADATA.contains(i.getAction()))
+        .limit(lookbackLimit)
+        .collect(Collectors.toList());
 
-      log.debug("Walking back up to {} commits to find rolling metadata for keys: {}",
-          lookbackLimit, remainingKeys);
+    log.debug("Walking back up to {} instants to find rolling metadata for keys: {}", lookbackLimit, remaining);
+    int instantsWalkedBack = 0;
 
-      for (HoodieInstant instant : recentCommits) {
-        if (remainingKeys.isEmpty()) {
-          break;  // Found all keys
+    try {
+      for (HoodieInstant instant : instants) {
+        if (remaining.isEmpty()) {
+          break;
         }
+        String action = instant.getAction();
+        Map<String, String> extraMeta = null;
 
-        commitsWalkedBack++;
-        HoodieCommitMetadata commitMetadata = table.getMetaClient().getActiveTimeline().readInstantContent(instant, HoodieCommitMetadata.class);
+        if (HoodieTimeline.CLEAN_ACTION.equals(action)) {
+          HoodieCleanMetadata cleanMeta = table.getActiveTimeline().readCleanMetadata(instant);
+          extraMeta = cleanMeta.getExtraMetadata();
+        } else {
+          HoodieCommitMetadata commitMeta = table.getMetaClient().getActiveTimeline()
+              .readInstantContent(instant, HoodieCommitMetadata.class);
+          extraMeta = commitMeta.getExtraMetadata();
+        }
+        instantsWalkedBack++;
 
-        // Check for remaining keys in this commit
-        for (String key : new HashSet<>(remainingKeys)) {
-          String value = commitMetadata.getMetadata(key);
+        if (extraMeta == null) {
+          continue;
+        }
+        for (String key : new HashSet<>(remaining)) {
+          String value = extraMeta.get(key);
           if (!StringUtils.isNullOrEmpty(value)) {
             foundRollingMetadata.put(key, value);
-            remainingKeys.remove(key);
-            log.debug("Found rolling metadata key '{}' in commit {} with value: {}",
-                key, instant.requestedTime(), value);
+            remaining.remove(key);
+            log.debug("Found rolling metadata key '{}' in {} instant {} with value: {}",
+                key, action, instant.requestedTime(), value);
           }
         }
       }
 
-      // Add found rolling metadata to current commit
-      for (Map.Entry<String, String> entry : foundRollingMetadata.entrySet()) {
-        metadata.addMetadata(entry.getKey(), entry.getValue());
-      }
-
-      int rolledForwardCount = foundRollingMetadata.size();
-      int updatedCount = rollingKeys.size() - remainingKeys.size() - rolledForwardCount;
-
-      if (rolledForwardCount > 0 || updatedCount > 0 || !remainingKeys.isEmpty()) {
-        log.info("Rolling metadata merge completed. Walked back {} commits. "
-                + "Rolled forward: {}, Updated in current: {}, Not found: {}, Total rolling keys: {}",
-            commitsWalkedBack, rolledForwardCount, updatedCount, remainingKeys.size(), rollingKeys.size());
+      if (!foundRollingMetadata.isEmpty() || !remaining.isEmpty()) {
+        log.info("Rolling metadata: walked {} instants. Rolled forward: {}, Not found: {}, Total keys: {}",
+            instantsWalkedBack, foundRollingMetadata.size(), remaining.size(), rollingKeys.size());
       }
-
-      if (!remainingKeys.isEmpty()) {
-        log.warn("Rolling metadata keys not found in last {} commits: {}. "
-            + "These keys will not be included in the current commit.", lookbackLimit, remainingKeys);
+      if (!remaining.isEmpty()) {
+        log.warn("Rolling metadata keys not found in last {} instants: {}.", instantsWalkedBack, remaining);
       }
-
     } catch (IOException e) {
-      log.error("Failed to read previous commit metadata for rolling metadata keys: {}.", rollingKeys, e);
-      throw new HoodieIOException("Failed to read previous commit metadata for rolling metadata keys: " + rollingKeys, e);
+      log.error("Failed to read previous metadata for rolling metadata keys: {}.", rollingKeys, e);
+      throw new HoodieIOException("Failed to read previous metadata for rolling keys: " + rollingKeys, e);
     }
+
+    return foundRollingMetadata;
   }
 }
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
index 5df834121bf9..a663c63dba78 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
@@ -768,21 +768,22 @@ public class HoodieWriteConfig extends HoodieConfig {
       .markAdvanced()
       .sinceVersion("1.2.0")
       .withDocumentation("Comma-separated list of extra metadata keys that should be automatically carried forward "
-          + "to every new commit. These keys will be read from recent commit metadata and included in new commits, "
-          + "ensuring they remain accessible without walking the timeline or worrying about archival. "
-          + "This is useful for tracking checkpoint information (e.g., Kafka offsets, Flink checkpoints) or any metadata "
-          + "that needs to persist across commits. New values override old ones. Only applies to data table commits.");
+          + "to every new commit and clean instant. These keys will be read from recent commit and clean metadata "
+          + "and included in new commits/cleans, ensuring they remain accessible without walking the timeline or "
+          + "worrying about archival. This is useful for tracking checkpoint information (e.g., Kafka offsets, "
+          + "Flink checkpoints) or any metadata that needs to persist across commits. New values override old ones. "
+          + "Only applies to data table commits and clean instants.");
 
   public static final ConfigProperty<Integer> ROLLING_METADATA_TIMELINE_LOOKBACK_COMMITS = ConfigProperty
       .key("hoodie.write.rolling.metadata.timeline.lookback.commits")
       .defaultValue(10)
       .markAdvanced()
       .sinceVersion("1.2.0")
-      .withDocumentation("Maximum number of completed commits to walk back in the timeline when searching for "
-          + "rolling metadata keys. If a rolling metadata key is not found in the latest commit, the system will "
-          + "walk back up to this many commits to find the most recent value. This ensures rolling metadata is "
-          + "preserved even if some commits don't update all keys. Higher values provide more resilience but may "
-          + "impact performance. Only applies when hoodie.write.rolling.metadata.keys is configured.");
+      .withDocumentation("Maximum number of completed instants (commits and clean) to walk back in the timeline "
+          + "when searching for rolling metadata keys. If a rolling metadata key is not found in the latest instant, "
+          + "the system will walk back up to this many instants to find the most recent value. This ensures rolling "
+          + "metadata is preserved even if some instants don't carry all keys. Higher values provide more resilience "
+          + "but may impact performance. Only applies when hoodie.write.rolling.metadata.keys is configured.");
 
   public static final ConfigProperty<Boolean> ALLOW_OPERATION_METADATA_FIELD = ConfigProperty
       .key("hoodie.allow.operation.metadata.field")
diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java
index 3fad30b22bdf..63a6e7d8f994 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java
@@ -21,6 +21,7 @@ package org.apache.hudi.table.action.clean;
 import org.apache.hudi.avro.model.HoodieActionInstant;
 import org.apache.hudi.avro.model.HoodieCleanMetadata;
 import org.apache.hudi.avro.model.HoodieCleanerPlan;
+import org.apache.hudi.client.BaseHoodieClient;
 import org.apache.hudi.client.transaction.TransactionManager;
 import org.apache.hudi.common.HoodieCleanStat;
 import org.apache.hudi.common.engine.HoodieEngineContext;
@@ -215,7 +216,6 @@ public class CleanActionExecutor<T, I, K, O> extends BaseActionExecutor<T, I, K,
       }
 
       List<HoodieCleanStat> cleanStats = clean(context, cleanerPlan);
-      table.getMetaClient().reloadActiveTimeline();
       HoodieCleanMetadata metadata;
       if (cleanStats.isEmpty()) {
         metadata = createEmptyCleanMetadata(cleanerPlan, inflightInstant, timer.endTimer());
@@ -228,6 +228,10 @@ public class CleanActionExecutor<T, I, K, O> extends BaseActionExecutor<T, I, K,
         );
       }
       this.txnManager.beginStateChange(Option.of(inflightInstant), Option.empty());
+      // Reload inside the lock so mergeRollingMetadata reads the latest timeline,
+      // matching the same contract as mergeRollingMetadata for commit metadata.
+      table.getMetaClient().reloadActiveTimeline();
+      BaseHoodieClient.mergeRollingMetadata(table, config, metadata);
       writeTableMetadata(metadata, inflightInstant.requestedTime());
       table.getActiveTimeline().transitionCleanInflightToComplete(
           false,
diff --git a/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java b/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java
index f9a59c6ab623..4ad8ec769ac3 100644
--- a/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java
+++ b/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java
@@ -30,6 +30,7 @@ import org.apache.hudi.avro.model.HoodieRollbackPlan;
 import org.apache.hudi.avro.model.HoodieSavepointMetadata;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
 import org.apache.hudi.common.table.timeline.versioning.TimelineLayoutVersion;
+import org.apache.hudi.common.util.CollectionUtils;
 import org.apache.hudi.common.util.Option;
 import org.apache.hudi.common.util.StringUtils;
 import org.apache.hudi.storage.HoodieInstantWriter;
@@ -77,6 +78,10 @@ public interface HoodieTimeline extends HoodieInstantReader, Serializable {
       CLEAN_ACTION, SAVEPOINT_ACTION, RESTORE_ACTION, ROLLBACK_ACTION,
       COMPACTION_ACTION, LOG_COMPACTION_ACTION, REPLACE_COMMIT_ACTION, CLUSTERING_ACTION, INDEXING_ACTION};
 
+  Set<String> VALID_ACTIONS_FOR_ROLLING_METADATA = CollectionUtils.createSet(
+      COMMIT_ACTION, DELTA_COMMIT_ACTION, CLEAN_ACTION,
+      COMPACTION_ACTION, LOG_COMPACTION_ACTION, REPLACE_COMMIT_ACTION, CLUSTERING_ACTION);
+
   String COMMIT_EXTENSION = "." + COMMIT_ACTION;
   String DELTA_COMMIT_EXTENSION = "." + DELTA_COMMIT_ACTION;
   String CLEAN_EXTENSION = "." + CLEAN_ACTION;
diff --git a/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/functional/TestHoodieClientOnCopyOnWriteStorage.java b/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/functional/TestHoodieClientOnCopyOnWriteStorage.java
index 2a4fa731e04b..ffb829165c7b 100644
--- a/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/functional/TestHoodieClientOnCopyOnWriteStorage.java
+++ b/hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/functional/TestHoodieClientOnCopyOnWriteStorage.java
@@ -18,6 +18,7 @@
 
 package org.apache.hudi.client.functional;
 
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
 import org.apache.hudi.avro.model.HoodieClusteringPlan;
 import org.apache.hudi.avro.model.HoodieRequestedReplaceMetadata;
 import org.apache.hudi.client.BaseHoodieWriteClient;
@@ -68,6 +69,7 @@ import org.apache.hudi.common.testutils.HoodieTestTable;
 import org.apache.hudi.common.util.ClusteringUtils;
 import org.apache.hudi.common.util.FileFormatUtils;
 import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.StringUtils;
 import org.apache.hudi.common.util.collection.Pair;
 import org.apache.hudi.config.HoodieArchivalConfig;
 import org.apache.hudi.config.HoodieCleanConfig;
@@ -2066,6 +2068,90 @@ public class TestHoodieClientOnCopyOnWriteStorage extends HoodieClientTestBase {
         "TableSchemaResolver should find schema even with clustering-only timeline");
   }
 
+  @Test
+  public void testRollingMetadataPreservedInCleanCommits() throws Exception {
+    String schemaKey = HoodieCommitMetadata.SCHEMA_KEY;
+    dataGen = new HoodieTestDataGenerator(new String[] {DEFAULT_FIRST_PARTITION_PATH});
+
+    HoodieWriteConfig config = getConfigBuilder(TRIP_EXAMPLE_SCHEMA)
+        .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+            .compactionSmallFileSize(0).build())
+        .withRollingMetadataKeys(schemaKey)
+        .withCleanConfig(HoodieCleanConfig.newBuilder()
+            .withAutoClean(false)
+            .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY)
+            .retainCommits(1)
+            .build())
+        .withArchivalConfig(HoodieArchivalConfig.newBuilder()
+            .archiveCommitsWith(10, 12).build())
+        .build();
+
+    SparkRDDWriteClient client = getHoodieWriteClient(config);
+
+    String firstCommit = client.startCommit();
+    List<HoodieRecord> records = dataGen.generateInserts(firstCommit, 100);
+    JavaRDD<WriteStatus> firstResult = client.insert(jsc.parallelize(records, 1), firstCommit);
+    client.commit(firstCommit, firstResult);
+
+    // Upsert several times to create superseded file versions
+    for (int i = 0; i < 4; i++) {
+      String commitTime = client.startCommit();
+      List<HoodieRecord> updates = dataGen.generateUpdates(commitTime, records);
+      JavaRDD<WriteStatus> result = client.upsert(jsc.parallelize(updates, 1), commitTime);
+      client.commit(commitTime, result);
+    }
+
+    // Run clean β€” retainCommits(1) means old file versions from earlier commits are eligible
+    HoodieCleanMetadata cleanResult = client.clean();
+    assertTrue(cleanResult != null, "Clean should produce metadata (files to clean exist)");
+
+    HoodieTableMetaClient freshMeta = HoodieTableMetaClient.reload(metaClient);
+    HoodieTimeline cleanTimeline = freshMeta.getActiveTimeline()
+        .getCleanerTimeline().filterCompletedInstants();
+    assertFalse(cleanTimeline.empty(), "Should have at least one clean instant");
+
+    HoodieInstant lastClean = cleanTimeline.lastInstant().get();
+    HoodieCleanMetadata cleanMetadata = cleanTimeline.readCleanMetadata(lastClean);
+
+    Map<String, String> cleanExtraMetadata = cleanMetadata.getExtraMetadata();
+    assertTrue(cleanExtraMetadata != null, "Clean metadata should have extraMetadata map");
+    assertTrue(cleanExtraMetadata.containsKey(schemaKey),
+        "Clean's extraMetadata should contain rolled-over schema key");
+    assertFalse(cleanExtraMetadata.get(schemaKey).isEmpty(),
+        "Rolled-over schema in clean should be non-empty");
+
+    // Now make one more commit with lookback=1. The rolling metadata walk should find
+    // the schema key in the clean instant (the most recent instant with the key).
+    HoodieWriteConfig lookback1Config = getConfigBuilder(TRIP_EXAMPLE_SCHEMA)
+        .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+            .compactionSmallFileSize(0).build())
+        .withRollingMetadataKeys(schemaKey)
+        .withRollingMetadataTimelineLookbackCommits(1)
+        .withCleanConfig(HoodieCleanConfig.newBuilder()
+            .withAutoClean(false)
+            .withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY)
+            .retainCommits(1)
+            .build())
+        .withArchivalConfig(HoodieArchivalConfig.newBuilder()
+            .archiveCommitsWith(10, 12).build())
+        .build();
+
+    SparkRDDWriteClient lookbackClient = getHoodieWriteClient(lookback1Config);
+    String nextCommit = lookbackClient.startCommit();
+    List<HoodieRecord> nextUpdates = dataGen.generateUpdates(nextCommit, records);
+    JavaRDD<WriteStatus> nextResult = lookbackClient.upsert(jsc.parallelize(nextUpdates, 1), nextCommit);
+    lookbackClient.commit(nextCommit, nextResult);
+
+    freshMeta = HoodieTableMetaClient.reload(metaClient);
+    HoodieTimeline commitsTimeline = freshMeta.getActiveTimeline()
+        .getCommitsTimeline().filterCompletedInstants();
+    HoodieInstant latestCommit = commitsTimeline.lastInstant().get();
+    HoodieCommitMetadata latestMeta = commitsTimeline.readCommitMetadata(latestCommit);
+    String rolledSchema = latestMeta.getMetadata(schemaKey);
+    assertFalse(StringUtils.isNullOrEmpty(rolledSchema),
+        "Schema should be rolled over into new commit even with lookback=1");
+  }
+
   /**
    * Disabling row writer here as clustering tests will throw the error below if it is used.
    * java.util.concurrent.CompletionException: java.lang.ClassNotFoundException


(hudi) 07/16: refactor(flink): Remove legacy Parquet nested readers superseded by Flink 2.1 Dremel path (FLINK-35702) (#18701)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit f0bf5eb7647dd88ee6a28f201f58a67651e88c88
Author: Shihuan Liu <sk...@gmail.com>
AuthorDate: Sat May 9 05:16:58 2026 -0700

    refactor(flink): Remove legacy Parquet nested readers superseded by Flink 2.1 Dremel path (FLINK-35702) (#18701)
    
    * refactor(flink): Remove legacy Parquet nested readers superseded by Flink 2.1 Dremel path (FLINK-35702)
    * Fix flaky IT test
    
    (cherry picked from commit 004d159968964d17875adb0ba97a51a206dfe1ff)
---
 .../apache/hudi/table/ITTestHoodieDataSource.java  |  42 +-
 .../format/cow/vector/ColumnarGroupArrayData.java  | 179 --------
 .../format/cow/vector/ColumnarGroupMapData.java    |  63 ---
 .../format/cow/vector/ColumnarGroupRowData.java    | 138 ------
 .../cow/vector/HeapArrayGroupColumnVector.java     |  53 ---
 .../cow/vector/reader/ArrayColumnReader.java       | 473 ---------------------
 .../format/cow/vector/reader/ArrayGroupReader.java |  44 --
 .../format/cow/vector/reader/MapColumnReader.java  |  56 ---
 .../cow/vector/reader/NestedColumnReader.java      |  15 +-
 .../vector/reader/NestedPrimitiveColumnReader.java |   4 +-
 .../format/cow/vector/reader/RowColumnReader.java  |  63 ---
 .../cow/vector/TestHeapColumnVectorAccessors.java  |   5 +-
 12 files changed, 53 insertions(+), 1082 deletions(-)

diff --git a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java
index 46b9353e0ec2..5d6a5daa061e 100644
--- a/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java
+++ b/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java
@@ -3555,11 +3555,51 @@ public class ITTestHoodieDataSource {
       // and max waiting timeout is 30s
       tableResult.await(30, TimeUnit.SECONDS);
     } catch (Throwable e) {
-      ExceptionUtils.assertThrowable(e, CollectSinkTableFactory.SuccessException.class);
+      // Acceptable terminal causes:
+      //   1. SuccessException: the sink reached its expected row count and intentionally
+      //      threw to terminate the streaming job. This is the happy path.
+      //   2. IOException("Stream is closed!") wrapped as HoodieIOException: a benign
+      //      error-attribution race between the source-side cascading-shutdown path and
+      //      the sink-side SuccessException terminator. When the sink throws
+      //      SuccessException to end the job, the chained source's SplitFetcher can close
+      //      the underlying Hadoop FSDataInputStream while the mailbox is still draining
+      //      a BatchRecords queued earlier; the next row-group read on the now-closed
+      //      stream surfaces an IOException("Stream is closed!"). With
+      //      restart-strategy.fixed-delay.attempts=0 (set in beforeEach to keep tests
+      //      deterministic) that IOException becomes the job's reported failure cause
+      //      instead of the sink's SuccessException, even though the sink has already
+      //      collected the expected rows by then - i.e. the functional outcome is
+      //      unchanged, only the error-attribution differs. Production paths correctly
+      //      fail the job on stream-closed-mid-read (the right behavior for real I/O
+      //      failures), so this tolerance is scoped to the SuccessException-based test
+      //      pattern below and is NOT mirrored in production code.
+      if (!isAcceptableTerminalFailure(e)) {
+        throw new AssertionError("Unexpected job failure", e);
+      }
     }
     tEnv.executeSql("DROP TABLE IF EXISTS sink");
     return CollectSinkTableFactory.RESULT.values().stream()
         .flatMap(Collection::stream)
         .collect(Collectors.toList());
   }
+
+  /**
+   * Whether {@code e} (or any of its causes) is one of the terminal failures that
+   * {@link #fetchResultWithExpectedNum} is allowed to swallow. See the comment at the call
+   * site for the rationale.
+   */
+  private static boolean isAcceptableTerminalFailure(Throwable e) {
+    Throwable cur = e;
+    while (cur != null) {
+      if (cur instanceof CollectSinkTableFactory.SuccessException) {
+        return true;
+      }
+      String msg = cur.getMessage();
+      if (msg != null && msg.contains("Stream is closed")) {
+        return true;
+      }
+      cur = cur.getCause();
+    }
+    return false;
+  }
 }
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/ColumnarGroupArrayData.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/ColumnarGroupArrayData.java
deleted file mode 100644
index 4c9275f3b093..000000000000
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/ColumnarGroupArrayData.java
+++ /dev/null
@@ -1,179 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.format.cow.vector;
-
-import org.apache.flink.table.data.ArrayData;
-import org.apache.flink.table.data.DecimalData;
-import org.apache.flink.table.data.MapData;
-import org.apache.flink.table.data.RawValueData;
-import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.data.StringData;
-import org.apache.flink.table.data.TimestampData;
-import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
-
-public class ColumnarGroupArrayData implements ArrayData {
-
-  WritableColumnVector vector;
-  int rowId;
-
-  public ColumnarGroupArrayData(WritableColumnVector vector, int rowId) {
-    this.vector = vector;
-    this.rowId = rowId;
-  }
-
-  @Override
-  public int size() {
-    if (vector == null) {
-      return 0;
-    }
-
-    if (vector instanceof HeapRowColumnVector) {
-      // assume all fields have the same size
-      if (((HeapRowColumnVector) vector).vectors == null || ((HeapRowColumnVector) vector).vectors.length == 0) {
-        return 0;
-      }
-      return ((HeapArrayVector) ((HeapRowColumnVector) vector).vectors[0]).getArray(rowId).size();
-    }
-    throw new UnsupportedOperationException(vector.getClass().getName() + " is not supported. Supported vector types: HeapRowColumnVector");
-  }
-
-  @Override
-  public boolean isNullAt(int index) {
-    if (vector == null) {
-      return true;
-    }
-
-    if (vector instanceof HeapRowColumnVector) {
-      return ((HeapRowColumnVector) vector).vectors == null;
-    }
-
-    throw new UnsupportedOperationException(vector.getClass().getName() + " is not supported. Supported vector types: HeapRowColumnVector");
-  }
-
-  @Override
-  public boolean getBoolean(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public byte getByte(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public short getShort(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public int getInt(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public long getLong(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public float getFloat(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public double getDouble(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public StringData getString(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public DecimalData getDecimal(int index, int precision, int scale) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public TimestampData getTimestamp(int index, int precision) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public <T> RawValueData<T> getRawValue(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public byte[] getBinary(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public ArrayData getArray(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public MapData getMap(int index) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public RowData getRow(int index, int numFields) {
-    return new ColumnarGroupRowData((HeapRowColumnVector) vector, rowId, index);
-  }
-
-  @Override
-  public boolean[] toBooleanArray() {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public byte[] toByteArray() {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public short[] toShortArray() {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public int[] toIntArray() {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public long[] toLongArray() {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public float[] toFloatArray() {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public double[] toDoubleArray() {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/ColumnarGroupMapData.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/ColumnarGroupMapData.java
deleted file mode 100644
index 69cb6feca13e..000000000000
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/ColumnarGroupMapData.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.format.cow.vector;
-
-import org.apache.flink.table.data.ArrayData;
-import org.apache.flink.table.data.MapData;
-import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
-
-public class ColumnarGroupMapData implements MapData {
-
-  WritableColumnVector keyVector;
-  WritableColumnVector valueVector;
-  int rowId;
-
-  public ColumnarGroupMapData(WritableColumnVector keyVector, WritableColumnVector valueVector, int rowId) {
-    this.keyVector = keyVector;
-    this.valueVector = valueVector;
-    this.rowId = rowId;
-  }
-
-  @Override
-  public int size() {
-    if (keyVector == null) {
-      return 0;
-    }
-
-    if (keyVector instanceof HeapArrayVector) {
-      return ((HeapArrayVector) keyVector).getArray(rowId).size();
-    }
-    throw new UnsupportedOperationException(keyVector.getClass().getName() + " is not supported. Supported vector types: HeapArrayVector");
-  }
-
-  @Override
-  public ArrayData keyArray() {
-    return ((HeapArrayVector) keyVector).getArray(rowId);
-  }
-
-  @Override
-  public ArrayData valueArray() {
-    if (valueVector instanceof HeapArrayVector) {
-      return ((HeapArrayVector) valueVector).getArray(rowId);
-    } else if (valueVector instanceof HeapArrayGroupColumnVector) {
-      return ((HeapArrayGroupColumnVector) valueVector).getArray(rowId);
-    }
-    throw new UnsupportedOperationException(valueVector.getClass().getName() + " is not supported. Supported vector types: HeapArrayVector, HeapArrayGroupColumnVector");
-  }
-}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/ColumnarGroupRowData.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/ColumnarGroupRowData.java
deleted file mode 100644
index 439c1880823f..000000000000
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/ColumnarGroupRowData.java
+++ /dev/null
@@ -1,138 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.format.cow.vector;
-
-import org.apache.flink.table.data.ArrayData;
-import org.apache.flink.table.data.DecimalData;
-import org.apache.flink.table.data.MapData;
-import org.apache.flink.table.data.RawValueData;
-import org.apache.flink.table.data.RowData;
-import org.apache.flink.table.data.StringData;
-import org.apache.flink.table.data.TimestampData;
-import org.apache.flink.types.RowKind;
-
-public class ColumnarGroupRowData implements RowData {
-
-  HeapRowColumnVector vector;
-  int rowId;
-  int index;
-
-  public ColumnarGroupRowData(HeapRowColumnVector vector, int rowId, int index) {
-    this.vector = vector;
-    this.rowId = rowId;
-    this.index = index;
-  }
-
-  @Override
-  public int getArity() {
-    return vector.vectors.length;
-  }
-
-  @Override
-  public RowKind getRowKind() {
-    return RowKind.INSERT;
-  }
-
-  @Override
-  public void setRowKind(RowKind rowKind) {
-    throw new UnsupportedOperationException("Not support the operation!");
-  }
-
-  @Override
-  public boolean isNullAt(int pos) {
-    return
-        vector.vectors[pos].isNullAt(rowId)
-            || ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).isNullAt(index);
-  }
-
-  @Override
-  public boolean getBoolean(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getBoolean(index);
-  }
-
-  @Override
-  public byte getByte(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getByte(index);
-  }
-
-  @Override
-  public short getShort(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getShort(index);
-  }
-
-  @Override
-  public int getInt(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getInt(index);
-  }
-
-  @Override
-  public long getLong(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getLong(index);
-  }
-
-  @Override
-  public float getFloat(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getFloat(index);
-  }
-
-  @Override
-  public double getDouble(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getDouble(index);
-  }
-
-  @Override
-  public StringData getString(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getString(index);
-  }
-
-  @Override
-  public DecimalData getDecimal(int pos, int i1, int i2) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getDecimal(index, i1, i2);
-  }
-
-  @Override
-  public TimestampData getTimestamp(int pos, int i1) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getTimestamp(index, i1);
-  }
-
-  @Override
-  public <T> RawValueData<T> getRawValue(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getRawValue(index);
-  }
-
-  @Override
-  public byte[] getBinary(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getBinary(index);
-  }
-
-  @Override
-  public ArrayData getArray(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getArray(index);
-  }
-
-  @Override
-  public MapData getMap(int pos) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getMap(index);
-  }
-
-  @Override
-  public RowData getRow(int pos, int numFields) {
-    return ((HeapArrayVector) (vector.vectors[pos])).getArray(rowId).getRow(index, numFields);
-  }
-}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapArrayGroupColumnVector.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapArrayGroupColumnVector.java
deleted file mode 100644
index 3d7d8b1f0de0..000000000000
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/HeapArrayGroupColumnVector.java
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.format.cow.vector;
-
-import org.apache.flink.table.data.ArrayData;
-import org.apache.flink.table.data.columnar.vector.ArrayColumnVector;
-import org.apache.flink.table.data.columnar.vector.heap.AbstractHeapVector;
-import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
-
-/**
- * This class represents a nullable heap row column vector.
- */
-public class HeapArrayGroupColumnVector extends AbstractHeapVector
-    implements WritableColumnVector, ArrayColumnVector {
-
-  public WritableColumnVector vector;
-
-  public HeapArrayGroupColumnVector(int len) {
-    super(len);
-  }
-
-  public HeapArrayGroupColumnVector(int len, WritableColumnVector vector) {
-    super(len);
-    this.vector = vector;
-  }
-
-  @Override
-  public ArrayData getArray(int rowId) {
-    return new ColumnarGroupArrayData(vector, rowId);
-  }
-
-  @Override
-  public void reset() {
-    super.reset();
-    vector.reset();
-  }
-}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ArrayColumnReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ArrayColumnReader.java
deleted file mode 100644
index d758f35078d8..000000000000
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ArrayColumnReader.java
+++ /dev/null
@@ -1,473 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.format.cow.vector.reader;
-
-import org.apache.hudi.table.format.cow.vector.HeapArrayVector;
-import org.apache.hudi.table.format.cow.vector.ParquetDecimalVector;
-
-import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
-import org.apache.flink.table.data.TimestampData;
-import org.apache.flink.table.data.columnar.vector.VectorizedColumnBatch;
-import org.apache.flink.table.data.columnar.vector.heap.HeapBooleanVector;
-import org.apache.flink.table.data.columnar.vector.heap.HeapByteVector;
-import org.apache.flink.table.data.columnar.vector.heap.HeapBytesVector;
-import org.apache.flink.table.data.columnar.vector.heap.HeapDoubleVector;
-import org.apache.flink.table.data.columnar.vector.heap.HeapFloatVector;
-import org.apache.flink.table.data.columnar.vector.heap.HeapIntVector;
-import org.apache.flink.table.data.columnar.vector.heap.HeapLongVector;
-import org.apache.flink.table.data.columnar.vector.heap.HeapShortVector;
-import org.apache.flink.table.data.columnar.vector.heap.HeapTimestampVector;
-import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
-import org.apache.flink.table.types.logical.ArrayType;
-import org.apache.flink.table.types.logical.LogicalType;
-import org.apache.parquet.column.ColumnDescriptor;
-import org.apache.parquet.column.page.PageReader;
-import org.apache.parquet.schema.PrimitiveType;
-import org.apache.parquet.schema.Type;
-
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-
-/**
- * Array {@link ColumnReader}.
- */
-public class ArrayColumnReader extends BaseVectorizedColumnReader {
-
-  // The value read in last time
-  private Object lastValue;
-
-  // flag to indicate if there is no data in parquet data page
-  private boolean eof = false;
-
-  // flag to indicate if it's the first time to read parquet data page with this instance
-  boolean isFirstRow = true;
-
-  public ArrayColumnReader(
-      ColumnDescriptor descriptor,
-      PageReader pageReader,
-      boolean isUtcTimestamp,
-      Type type,
-      LogicalType logicalType)
-      throws IOException {
-    super(descriptor, pageReader, isUtcTimestamp, type, logicalType);
-  }
-
-  @Override
-  public void readToVector(int readNumber, WritableColumnVector vector) throws IOException {
-    HeapArrayVector lcv = (HeapArrayVector) vector;
-    // before readBatch, initial the size of offsets & lengths as the default value,
-    // the actual size will be assigned in setChildrenInfo() after reading complete.
-    lcv.offsets = new long[VectorizedColumnBatch.DEFAULT_SIZE];
-    lcv.lengths = new long[VectorizedColumnBatch.DEFAULT_SIZE];
-    // Because the length of ListColumnVector.child can't be known now,
-    // the valueList will save all data for ListColumnVector temporary.
-    List<Object> valueList = new ArrayList<>();
-
-    LogicalType category = ((ArrayType) logicalType).getElementType();
-
-    // read the first row in parquet data page, this will be only happened once for this
-    // instance
-    if (isFirstRow) {
-      if (!fetchNextValue(category)) {
-        return;
-      }
-      isFirstRow = false;
-    }
-
-    int index = collectDataFromParquetPage(readNumber, lcv, valueList, category);
-
-    // Convert valueList to array for the ListColumnVector.child
-    fillColumnVector(category, lcv, valueList, index);
-  }
-
-  /**
-   * Reads a single value from parquet page, puts it into lastValue. Returns a boolean indicating
-   * if there is more values to read (true).
-   *
-   * @param category
-   * @return boolean
-   * @throws IOException
-   */
-  private boolean fetchNextValue(LogicalType category) throws IOException {
-    int left = readPageIfNeed();
-    if (left > 0) {
-      // get the values of repetition and definitionLevel
-      readRepetitionAndDefinitionLevels();
-      // read the data if it isn't null
-      if (definitionLevel == maxDefLevel) {
-        if (isCurrentPageDictionaryEncoded) {
-          lastValue = dataColumn.readValueDictionaryId();
-        } else {
-          lastValue = readPrimitiveTypedRow(category);
-        }
-      } else {
-        lastValue = null;
-      }
-      return true;
-    } else {
-      eof = true;
-      return false;
-    }
-  }
-
-  private int readPageIfNeed() throws IOException {
-    // Compute the number of values we want to read in this page.
-    int leftInPage = (int) (endOfPageValueCount - valuesRead);
-    if (leftInPage == 0) {
-      // no data left in current page, load data from new page
-      readPage();
-      leftInPage = (int) (endOfPageValueCount - valuesRead);
-    }
-    return leftInPage;
-  }
-
-  // Need to be in consistent with that VectorizedPrimitiveColumnReader#readBatchHelper
-  // TODO Reduce the duplicated code
-  private Object readPrimitiveTypedRow(LogicalType category) {
-    switch (category.getTypeRoot()) {
-      case CHAR:
-      case VARCHAR:
-      case BINARY:
-      case VARBINARY:
-        return dataColumn.readString();
-      case BOOLEAN:
-        return dataColumn.readBoolean();
-      case TIME_WITHOUT_TIME_ZONE:
-      case DATE:
-      case INTEGER:
-        return dataColumn.readInteger();
-      case TINYINT:
-        return dataColumn.readTinyInt();
-      case SMALLINT:
-        return dataColumn.readSmallInt();
-      case BIGINT:
-        return dataColumn.readLong();
-      case FLOAT:
-        return dataColumn.readFloat();
-      case DOUBLE:
-        return dataColumn.readDouble();
-      case DECIMAL:
-        switch (descriptor.getPrimitiveType().getPrimitiveTypeName()) {
-          case INT32:
-            return dataColumn.readInteger();
-          case INT64:
-            return dataColumn.readLong();
-          case BINARY:
-          case FIXED_LEN_BYTE_ARRAY:
-            return dataColumn.readString();
-          default:
-            throw new AssertionError();
-        }
-      case TIMESTAMP_WITHOUT_TIME_ZONE:
-      case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
-        return dataColumn.readTimestamp();
-      default:
-        throw new RuntimeException("Unsupported type in the list: " + type);
-    }
-  }
-
-  private Object dictionaryDecodeValue(LogicalType category, Integer dictionaryValue) {
-    if (dictionaryValue == null) {
-      return null;
-    }
-
-    switch (category.getTypeRoot()) {
-      case CHAR:
-      case VARCHAR:
-      case BINARY:
-      case VARBINARY:
-        return dictionary.readString(dictionaryValue);
-      case DATE:
-      case TIME_WITHOUT_TIME_ZONE:
-      case INTEGER:
-        return dictionary.readInteger(dictionaryValue);
-      case BOOLEAN:
-        return dictionary.readBoolean(dictionaryValue) ? 1 : 0;
-      case DOUBLE:
-        return dictionary.readDouble(dictionaryValue);
-      case FLOAT:
-        return dictionary.readFloat(dictionaryValue);
-      case TINYINT:
-        return dictionary.readTinyInt(dictionaryValue);
-      case SMALLINT:
-        return dictionary.readSmallInt(dictionaryValue);
-      case BIGINT:
-        return dictionary.readLong(dictionaryValue);
-      case DECIMAL:
-        switch (descriptor.getPrimitiveType().getPrimitiveTypeName()) {
-          case INT32:
-            return dictionary.readInteger(dictionaryValue);
-          case INT64:
-            return dictionary.readLong(dictionaryValue);
-          case FIXED_LEN_BYTE_ARRAY:
-          case BINARY:
-            return dictionary.readString(dictionaryValue);
-          default:
-            throw new AssertionError();
-        }
-      case TIMESTAMP_WITHOUT_TIME_ZONE:
-      case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
-        return dictionary.readTimestamp(dictionaryValue);
-      default:
-        throw new RuntimeException("Unsupported type in the list: " + type);
-    }
-  }
-
-  /**
-   * Collects data from a parquet page and returns the final row index where it stopped. The
-   * returned index can be equal to or less than total.
-   *
-   * @param total     maximum number of rows to collect
-   * @param lcv       column vector to do initial setup in data collection time
-   * @param valueList collection of values that will be fed into the vector later
-   * @param category
-   * @return int
-   * @throws IOException
-   */
-  private int collectDataFromParquetPage(
-      int total, HeapArrayVector lcv, List<Object> valueList, LogicalType category)
-      throws IOException {
-    int index = 0;
-    /*
-     * Here is a nested loop for collecting all values from a parquet page.
-     * A column of array type can be considered as a list of lists, so the two loops are as below:
-     * 1. The outer loop iterates on rows (index is a row index, so points to a row in the batch), e.g.:
-     * [0, 2, 3]    <- index: 0
-     * [NULL, 3, 4] <- index: 1
-     *
-     * 2. The inner loop iterates on values within a row (sets all data from parquet data page
-     * for an element in ListColumnVector), so fetchNextValue returns values one-by-one:
-     * 0, 2, 3, NULL, 3, 4
-     *
-     * As described below, the repetition level (repetitionLevel != 0)
-     * can be used to decide when we'll start to read values for the next list.
-     */
-    while (!eof && index < total) {
-      // add element to ListColumnVector one by one
-      lcv.offsets[index] = valueList.size();
-      /*
-       * Let's collect all values for a single list.
-       * Repetition level = 0 means that a new list started there in the parquet page,
-       * in that case, let's exit from the loop, and start to collect value for a new list.
-       */
-      do {
-        /*
-         * Definition level = 0 when a NULL value was returned instead of a list
-         * (this is not the same as a NULL value in of a list).
-         */
-        if (definitionLevel == 0) {
-          lcv.setNullAt(index);
-        }
-        valueList.add(
-            isCurrentPageDictionaryEncoded
-                ? dictionaryDecodeValue(category, (Integer) lastValue)
-                : lastValue);
-      } while (fetchNextValue(category) && (repetitionLevel != 0));
-
-      lcv.lengths[index] = valueList.size() - lcv.offsets[index];
-      index++;
-    }
-    return index;
-  }
-
-  /**
-   * The lengths & offsets will be initialized as default size (1024), it should be set to the
-   * actual size according to the element number.
-   */
-  private void setChildrenInfo(HeapArrayVector lcv, int itemNum, int elementNum) {
-    lcv.setSize(itemNum);
-    long[] lcvLength = new long[elementNum];
-    long[] lcvOffset = new long[elementNum];
-    System.arraycopy(lcv.lengths, 0, lcvLength, 0, elementNum);
-    System.arraycopy(lcv.offsets, 0, lcvOffset, 0, elementNum);
-    lcv.lengths = lcvLength;
-    lcv.offsets = lcvOffset;
-  }
-
-  private void fillColumnVector(
-      LogicalType category, HeapArrayVector lcv, List valueList, int elementNum) {
-    int total = valueList.size();
-    setChildrenInfo(lcv, total, elementNum);
-    switch (category.getTypeRoot()) {
-      case CHAR:
-      case VARCHAR:
-      case BINARY:
-      case VARBINARY:
-        lcv.child = new HeapBytesVector(total);
-        ((HeapBytesVector) lcv.child).reset();
-        for (int i = 0; i < valueList.size(); i++) {
-          byte[] src = ((List<byte[]>) valueList).get(i);
-          if (src == null) {
-            ((HeapBytesVector) lcv.child).setNullAt(i);
-          } else {
-            ((HeapBytesVector) lcv.child).appendBytes(i, src, 0, src.length);
-          }
-        }
-        break;
-      case BOOLEAN:
-        lcv.child = new HeapBooleanVector(total);
-        ((HeapBooleanVector) lcv.child).reset();
-        for (int i = 0; i < valueList.size(); i++) {
-          if (valueList.get(i) == null) {
-            ((HeapBooleanVector) lcv.child).setNullAt(i);
-          } else {
-            ((HeapBooleanVector) lcv.child).vector[i] =
-                ((List<Boolean>) valueList).get(i);
-          }
-        }
-        break;
-      case TINYINT:
-        lcv.child = new HeapByteVector(total);
-        ((HeapByteVector) lcv.child).reset();
-        for (int i = 0; i < valueList.size(); i++) {
-          if (valueList.get(i) == null) {
-            ((HeapByteVector) lcv.child).setNullAt(i);
-          } else {
-            ((HeapByteVector) lcv.child).vector[i] =
-                (byte) ((List<Integer>) valueList).get(i).intValue();
-          }
-        }
-        break;
-      case SMALLINT:
-        lcv.child = new HeapShortVector(total);
-        ((HeapShortVector) lcv.child).reset();
-        for (int i = 0; i < valueList.size(); i++) {
-          if (valueList.get(i) == null) {
-            ((HeapShortVector) lcv.child).setNullAt(i);
-          } else {
-            ((HeapShortVector) lcv.child).vector[i] =
-                (short) ((List<Integer>) valueList).get(i).intValue();
-          }
-        }
-        break;
-      case INTEGER:
-      case DATE:
-      case TIME_WITHOUT_TIME_ZONE:
-        lcv.child = new HeapIntVector(total);
-        ((HeapIntVector) lcv.child).reset();
-        for (int i = 0; i < valueList.size(); i++) {
-          if (valueList.get(i) == null) {
-            ((HeapIntVector) lcv.child).setNullAt(i);
-          } else {
-            ((HeapIntVector) lcv.child).vector[i] = ((List<Integer>) valueList).get(i);
-          }
-        }
-        break;
-      case FLOAT:
-        lcv.child = new HeapFloatVector(total);
-        ((HeapFloatVector) lcv.child).reset();
-        for (int i = 0; i < valueList.size(); i++) {
-          if (valueList.get(i) == null) {
-            ((HeapFloatVector) lcv.child).setNullAt(i);
-          } else {
-            ((HeapFloatVector) lcv.child).vector[i] = ((List<Float>) valueList).get(i);
-          }
-        }
-        break;
-      case BIGINT:
-        lcv.child = new HeapLongVector(total);
-        ((HeapLongVector) lcv.child).reset();
-        for (int i = 0; i < valueList.size(); i++) {
-          if (valueList.get(i) == null) {
-            ((HeapLongVector) lcv.child).setNullAt(i);
-          } else {
-            ((HeapLongVector) lcv.child).vector[i] = ((List<Long>) valueList).get(i);
-          }
-        }
-        break;
-      case DOUBLE:
-        lcv.child = new HeapDoubleVector(total);
-        ((HeapDoubleVector) lcv.child).reset();
-        for (int i = 0; i < valueList.size(); i++) {
-          if (valueList.get(i) == null) {
-            ((HeapDoubleVector) lcv.child).setNullAt(i);
-          } else {
-            ((HeapDoubleVector) lcv.child).vector[i] =
-                ((List<Double>) valueList).get(i);
-          }
-        }
-        break;
-      case TIMESTAMP_WITHOUT_TIME_ZONE:
-      case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
-        lcv.child = new HeapTimestampVector(total);
-        ((HeapTimestampVector) lcv.child).reset();
-        for (int i = 0; i < valueList.size(); i++) {
-          if (valueList.get(i) == null) {
-            ((HeapTimestampVector) lcv.child).setNullAt(i);
-          } else {
-            ((HeapTimestampVector) lcv.child)
-                .setTimestamp(i, ((List<TimestampData>) valueList).get(i));
-          }
-        }
-        break;
-      case DECIMAL:
-        PrimitiveType.PrimitiveTypeName primitiveTypeName =
-            descriptor.getPrimitiveType().getPrimitiveTypeName();
-        switch (primitiveTypeName) {
-          case INT32:
-            lcv.child = new ParquetDecimalVector(new HeapIntVector(total));
-            ((HeapIntVector) ((ParquetDecimalVector) lcv.child).getVector()).reset();
-            for (int i = 0; i < valueList.size(); i++) {
-              if (valueList.get(i) == null) {
-                ((HeapIntVector) ((ParquetDecimalVector) lcv.child).getVector())
-                    .setNullAt(i);
-              } else {
-                ((HeapIntVector) ((ParquetDecimalVector) lcv.child).getVector())
-                    .vector[i] =
-                    ((List<Integer>) valueList).get(i);
-              }
-            }
-            break;
-          case INT64:
-            lcv.child = new ParquetDecimalVector(new HeapLongVector(total));
-            ((HeapLongVector) ((ParquetDecimalVector) lcv.child).getVector()).reset();
-            for (int i = 0; i < valueList.size(); i++) {
-              if (valueList.get(i) == null) {
-                ((HeapLongVector) ((ParquetDecimalVector) lcv.child).getVector())
-                    .setNullAt(i);
-              } else {
-                ((HeapLongVector) ((ParquetDecimalVector) lcv.child).getVector())
-                    .vector[i] =
-                    ((List<Long>) valueList).get(i);
-              }
-            }
-            break;
-          default:
-            lcv.child = new ParquetDecimalVector(new HeapBytesVector(total));
-            ((HeapBytesVector) ((ParquetDecimalVector) lcv.child).getVector()).reset();
-            for (int i = 0; i < valueList.size(); i++) {
-              byte[] src = ((List<byte[]>) valueList).get(i);
-              if (valueList.get(i) == null) {
-                ((HeapBytesVector) ((ParquetDecimalVector) lcv.child).getVector())
-                    .setNullAt(i);
-              } else {
-                ((HeapBytesVector) ((ParquetDecimalVector) lcv.child).getVector())
-                    .appendBytes(i, src, 0, src.length);
-              }
-            }
-            break;
-        }
-        break;
-      default:
-        throw new RuntimeException("Unsupported type in the list: " + type);
-    }
-  }
-}
-
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ArrayGroupReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ArrayGroupReader.java
deleted file mode 100644
index 437c186a9366..000000000000
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/ArrayGroupReader.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.format.cow.vector.reader;
-
-import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
-import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
-import org.apache.hudi.table.format.cow.vector.HeapArrayGroupColumnVector;
-
-import java.io.IOException;
-
-/**
- * Array of a Group type (Array, Map, Row, etc.) {@link ColumnReader}.
- */
-public class ArrayGroupReader implements ColumnReader<WritableColumnVector> {
-
-  private final ColumnReader<WritableColumnVector> fieldReader;
-
-  public ArrayGroupReader(ColumnReader<WritableColumnVector> fieldReader) {
-    this.fieldReader = fieldReader;
-  }
-
-  @Override
-  public void readToVector(int readNumber, WritableColumnVector vector) throws IOException {
-    HeapArrayGroupColumnVector rowColumnVector = (HeapArrayGroupColumnVector) vector;
-
-    fieldReader.readToVector(readNumber, rowColumnVector.vector);
-  }
-}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/MapColumnReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/MapColumnReader.java
deleted file mode 100644
index ee65dd22c436..000000000000
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/MapColumnReader.java
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.format.cow.vector.reader;
-
-import org.apache.hudi.table.format.cow.vector.HeapMapColumnVector;
-
-import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
-import org.apache.flink.table.data.columnar.vector.heap.AbstractHeapVector;
-import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
-
-import java.io.IOException;
-
-/**
- * Map {@link ColumnReader}.
- */
-public class MapColumnReader implements ColumnReader<WritableColumnVector> {
-
-  private final ArrayColumnReader keyReader;
-  private final ColumnReader<WritableColumnVector> valueReader;
-
-  public MapColumnReader(
-      ArrayColumnReader keyReader, ColumnReader<WritableColumnVector> valueReader) {
-    this.keyReader = keyReader;
-    this.valueReader = valueReader;
-  }
-
-  @Override
-  public void readToVector(int readNumber, WritableColumnVector vector) throws IOException {
-    HeapMapColumnVector mapColumnVector = (HeapMapColumnVector) vector;
-    AbstractHeapVector keyArrayColumnVector = (AbstractHeapVector) (mapColumnVector.getKeys());
-    keyReader.readToVector(readNumber, mapColumnVector.getKeys());
-    valueReader.readToVector(readNumber, mapColumnVector.getValues());
-    for (int i = 0; i < keyArrayColumnVector.getLen(); i++) {
-      if (keyArrayColumnVector.isNullAt(i)) {
-        mapColumnVector.setNullAt(i);
-      }
-    }
-  }
-}
-
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java
index 6ae1cc9492ec..b4f04b20c477 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedColumnReader.java
@@ -114,9 +114,8 @@ public class NestedColumnReader implements ColumnReader<WritableColumnVector> {
       ParquetField child = children.get(i);
       if (child == null) {
         // Hudi schema-evolution: the logical field is not present in the Parquet file. The slot
-        // vector is expected to be pre-populated with nulls by the caller (lands in a follow-up
-        // PR that rewires ParquetSplitReaderUtil); keep it as is and skip contributing to the
-        // level stream.
+        // vector was pre-populated with nulls by ParquetSplitReaderUtil#createWritableColumnVector
+        // (ROW branch); keep it as is and skip contributing to the level stream.
         finalChildrenVectors[i] = childrenVectors[i];
         continue;
       }
@@ -148,11 +147,11 @@ public class NestedColumnReader implements ColumnReader<WritableColumnVector> {
       setFieldNullFlag(rowPosition.getIsNull(), heapRowVector);
     }
 
-    // Hudi-specific: collapse a present row whose every child is null into a null row. The
-    // legacy RowColumnReader did this so that a SQL value like `row(null, null)` round-trips
-    // to NULL on read; preserve it here for backward compatibility. Diverges from Flink 2.1,
-    // which would surface it as Row(null, null). Mirrored by the integration test
-    // ITTestHoodieDataSource#testParquetNullChildColumnsRowTypes.
+    // Hudi-specific: collapse a present row whose every child is null into a null row, so that a
+    // SQL value like `row(null, null)` round-trips to NULL on read. This was the behaviour of the
+    // legacy RowColumnReader (deleted alongside the Dremel rewire) and existing Hudi tables rely
+    // on it. Diverges from Flink 2.1, which would surface it as Row(null, null). Pinned by the
+    // integration test ITTestHoodieDataSource#testParquetNullChildColumnsRowTypes.
     int rowCount = rowPosition.getPositionsCount();
     for (int j = 0; j < rowCount; j++) {
       if (heapRowVector.isNullAt(j)) {
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedPrimitiveColumnReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedPrimitiveColumnReader.java
index 3f0aefe2af74..72809db1b2ce 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedPrimitiveColumnReader.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/NestedPrimitiveColumnReader.java
@@ -71,8 +71,8 @@ import static org.apache.parquet.column.ValuesType.VALUES;
  * and the Hudi-local {@link ParquetDecimalVector} / {@link LevelDelegation} / {@link IntArrayList}
  * imports are changed; the algorithm is untouched. The companion Hudi-specific {@code
  * Int64TimestampColumnReader} / {@code FixedLenBytesColumnReader} behaviours stay at the leaf-
- * reader creation boundary in {@code ParquetSplitReaderUtil} (lands in a follow-up PR), not inside
- * this class β€” keeping it a faithful copy of upstream.
+ * reader creation boundary in {@code ParquetSplitReaderUtil}, not inside this class β€” keeping it
+ * a faithful copy of upstream.
  */
 public class NestedPrimitiveColumnReader implements ColumnReader<WritableColumnVector> {
   private static final Logger LOG = LoggerFactory.getLogger(NestedPrimitiveColumnReader.class);
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/RowColumnReader.java b/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/RowColumnReader.java
deleted file mode 100644
index 79b50487f13c..000000000000
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/main/java/org/apache/hudi/table/format/cow/vector/reader/RowColumnReader.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.format.cow.vector.reader;
-
-import org.apache.hudi.table.format.cow.vector.HeapRowColumnVector;
-
-import org.apache.flink.formats.parquet.vector.reader.ColumnReader;
-import org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector;
-
-import java.io.IOException;
-import java.util.List;
-
-/**
- * Row {@link ColumnReader}.
- */
-public class RowColumnReader implements ColumnReader<WritableColumnVector> {
-
-  private final List<ColumnReader> fieldReaders;
-
-  public RowColumnReader(List<ColumnReader> fieldReaders) {
-    this.fieldReaders = fieldReaders;
-  }
-
-  @Override
-  public void readToVector(int readNumber, WritableColumnVector vector) throws IOException {
-    HeapRowColumnVector rowColumnVector = (HeapRowColumnVector) vector;
-    WritableColumnVector[] vectors = rowColumnVector.vectors;
-    // row vector null array
-    boolean[] isNulls = new boolean[readNumber];
-    for (int i = 0; i < vectors.length; i++) {
-      fieldReaders.get(i).readToVector(readNumber, vectors[i]);
-
-      for (int j = 0; j < readNumber; j++) {
-        if (i == 0) {
-          isNulls[j] = vectors[i].isNullAt(j);
-        } else {
-          isNulls[j] = isNulls[j] && vectors[i].isNullAt(j);
-        }
-        if (i == vectors.length - 1 && isNulls[j]) {
-          // rowColumnVector[j] is null only when all fields[j] of rowColumnVector[j] is
-          // null
-          rowColumnVector.setNullAt(j);
-        }
-      }
-    }
-  }
-}
diff --git a/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/TestHeapColumnVectorAccessors.java b/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/TestHeapColumnVectorAccessors.java
index 4b48fdd37460..7cb62824e854 100644
--- a/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/TestHeapColumnVectorAccessors.java
+++ b/hudi-flink-datasource/hudi-flink1.18.x/src/test/java/org/apache/hudi/table/format/cow/vector/TestHeapColumnVectorAccessors.java
@@ -35,8 +35,9 @@ import static org.junit.jupiter.api.Assertions.assertSame;
  *
  * <p>The accessors are wrappers over the existing public fields so legacy callers continue to
  * work. These tests exist solely to pin down that wrapper contract β€” runtime correctness of the
- * Dremel-style read path is exercised end-to-end by integration tests once
- * {@code ParquetSplitReaderUtil} is wired up to {@code NestedColumnReader} in a follow-up PR.
+ * Dremel-style read path is exercised end-to-end by integration tests in
+ * {@code ITTestHoodieDataSource} (testParquetComplexTypes / testParquetComplexNestedRowTypes /
+ * testParquetArrayMapOfRowTypes / testParquetNullChildColumnsRowTypes).
  */
 class TestHeapColumnVectorAccessors {
 


(hudi) 01/16: chore(docker): reduce base_java17 and spark_base image size (#18542)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 9faeca0351bc52b7c1fb9986bc4a0823307e59ec
Author: kartikeyaagrawal <52...@users.noreply.github.com>
AuthorDate: Thu Apr 30 15:10:08 2026 +0530

    chore(docker): reduce base_java17 and spark_base image size (#18542)
    
    Addresses #18523.
    
    Shrinks the Java 17 integration-test image from ~3.56 GB to ~2.58 GB
    (~27%) without changing any runtime behavior. The container commands,
    environment variables, exposed ports, and entrypoint are identical.
    
    base_java17/Dockerfile:
    - switch base image from eclipse-temurin:17-jdk to 17-jre-jammy. The
      container only runs Hadoop; no Java compilation happens inside it,
      so the JDK toolchain is not needed.
    - convert to a multi-stage build. Stage 1 downloads and extracts the
      Hadoop tarball; stage 2 only COPYs the extracted tree. curl,
      ca-certificates, and the tar.gz no longer land in the final layer.
    - use --no-install-recommends and clean apt lists in the runtime stage.
    - drop the unused .asc signature download and the now-dead wget dep.
    
    spark_base/Dockerfile:
    - replace the Python-3.10.14-from-source build (which pulled in
      build-essential and a full compile toolchain, then built CPython
      with --enable-optimizations inside the image) with the distro
      python3-minimal + python3-pip packages. PySpark only needs a
      Python runtime at runtime.
    
    (cherry picked from commit 0a28d695eb1558b01bb38b11996a436a2e3e23dd)
---
 docker/hoodie/hadoop/base_java17/Dockerfile | 41 ++++++++++++++++++-----------
 docker/hoodie/hadoop/spark_base/Dockerfile  | 21 +++------------
 2 files changed, 30 insertions(+), 32 deletions(-)

diff --git a/docker/hoodie/hadoop/base_java17/Dockerfile b/docker/hoodie/hadoop/base_java17/Dockerfile
index 45108610b19e..a5c265d3aaac 100644
--- a/docker/hoodie/hadoop/base_java17/Dockerfile
+++ b/docker/hoodie/hadoop/base_java17/Dockerfile
@@ -15,7 +15,23 @@
 #  See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM eclipse-temurin:17-jdk
+# --- Stage 1: fetch + extract Hadoop (throwaway) ---
+FROM eclipse-temurin:17-jre-jammy AS hadoop-builder
+
+ARG HADOOP_VERSION=3.4.0
+ARG HADOOP_URL=https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz
+
+RUN set -x \
+    && DEBIAN_FRONTEND=noninteractive apt-get -yq update \
+    && apt-get -yq install --no-install-recommends curl ca-certificates \
+    && curl -fSL "${HADOOP_URL}" -o /tmp/hadoop.tar.gz \
+    && mkdir -p /opt \
+    && tar -xzf /tmp/hadoop.tar.gz -C /opt/ \
+    && rm /tmp/hadoop.tar.gz \
+    && mkdir -p /opt/hadoop-${HADOOP_VERSION}/logs
+
+# --- Stage 2: runtime ---
+FROM eclipse-temurin:17-jre-jammy
 LABEL maintainer="Hoodie"
 USER root
 
@@ -23,28 +39,23 @@ USER root
 ENV LANG C.UTF-8
 
 ARG HADOOP_VERSION=3.4.0
-ARG HADOOP_URL=https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz
-ENV HADOOP_VERSION ${HADOOP_VERSION}
-ENV HADOOP_URL ${HADOOP_URL}
+ENV HADOOP_VERSION=${HADOOP_VERSION}
 
-RUN set -x \
-    && DEBIAN_FRONTEND=noninteractive apt-get -yq update && apt-get -yq install curl wget netcat-openbsd procps \
-    && echo "Fetch URL2 is : ${HADOOP_URL}" \
-    && curl -fSL "${HADOOP_URL}" -o /tmp/hadoop.tar.gz \
-    && curl -fSL "${HADOOP_URL}.asc" -o /tmp/hadoop.tar.gz.asc \
-    && mkdir -p /opt/hadoop-$HADOOP_VERSION/logs \
-    && tar -xvf /tmp/hadoop.tar.gz -C /opt/ \
-    && rm /tmp/hadoop.tar.gz* \
-    && ln -s /opt/hadoop-$HADOOP_VERSION/etc/hadoop /etc/hadoop \
+RUN DEBIAN_FRONTEND=noninteractive apt-get -yq update \
+    && apt-get -yq install --no-install-recommends netcat-openbsd procps \
+    && rm -rf /var/lib/apt/lists/* \
     && mkdir /hadoop-data
 
-ENV HADOOP_PREFIX=/opt/hadoop-$HADOOP_VERSION
+COPY --from=hadoop-builder /opt/hadoop-${HADOOP_VERSION} /opt/hadoop-${HADOOP_VERSION}
+RUN ln -s /opt/hadoop-${HADOOP_VERSION}/etc/hadoop /etc/hadoop
+
+ENV HADOOP_PREFIX=/opt/hadoop-${HADOOP_VERSION}
 ENV HADOOP_CONF_DIR=/etc/hadoop
 ENV MULTIHOMED_NETWORK=1
 ENV HADOOP_HOME=${HADOOP_PREFIX}
 ENV HADOOP_INSTALL=${HADOOP_HOME}
 ENV USER=root
-ENV PATH /usr/bin:/bin:$HADOOP_PREFIX/bin/:$PATH
+ENV PATH=/usr/bin:/bin:${HADOOP_PREFIX}/bin/:$PATH
 
 # Exposing a union of ports across hadoop versions
 # Well known ports including ssh
diff --git a/docker/hoodie/hadoop/spark_base/Dockerfile b/docker/hoodie/hadoop/spark_base/Dockerfile
index 68bfbaae76d8..4210faf21bfe 100644
--- a/docker/hoodie/hadoop/spark_base/Dockerfile
+++ b/docker/hoodie/hadoop/spark_base/Dockerfile
@@ -41,23 +41,10 @@ RUN echo "Installing Spark-version (${SPARK_VERSION})" \
       && rm spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz \
       && cd /
 
-# Install build dependencies
-RUN apt-get update && apt-get install -y \
-    wget build-essential libncursesw5-dev \
-    libssl-dev libgdbm-dev libreadline-dev libbz2-dev \
-    libsqlite3-dev libffi-dev zlib1g-dev curl \
-    && cd /usr/src \
-    && wget https://www.python.org/ftp/python/3.10.14/Python-3.10.14.tgz \
-    && tar xzf Python-3.10.14.tgz \
-    && cd Python-3.10.14 \
-    && ./configure --enable-optimizations \
-    && make -j"$(nproc)" \
-    && make altinstall \
-    && ln -sf /usr/local/bin/python3.10 /usr/bin/python \
-    && ln -sf /usr/local/bin/python3.10 /usr/bin/python3 \
-    && curl -sS https://bootstrap.pypa.io/get-pip.py | python \
-    && pip install --upgrade pip \
-    && cd / && rm -rf /usr/src/Python-3.10.14* \
+# Install Python runtime from distro package (avoids ~400MB build toolchain)
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends python3-minimal python3-pip \
+    && ln -sf /usr/bin/python3 /usr/bin/python \
     && rm -rf /var/lib/apt/lists/*
 
 #Give permission to execute scripts


(hudi) 16/16: feat(spark): add restore_to_instant stored procedure (#18696)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit dbfafbcf5738ce7c34c844492145530359c7a8ea
Author: Mahsood Ebrahim <ma...@gmail.com>
AuthorDate: Mon May 18 23:43:55 2026 -0700

    feat(spark): add restore_to_instant stored procedure (#18696)
    
    * feat(spark): add restore_to_instant stored procedure
    
    Adds a Spark SQL stored procedure that performs a point-in-time table
    restore to any instant on the active timeline, with optional post-restore
    file-existence audit. Unlike rollback_to_savepoint, no savepoint is
    required at the target instant.
    
    Centralizes the MDT pre-check that was inlined in restoreToSavepoint into
    a new BaseHoodieWriteClient.shouldDeleteMdtBeforeRestore helper, and
    extends restoreToInstant to invoke it. The helper also catches the
    penultimate-compaction case (target at or before the second-most-recent
    MDT compaction) which the previous restoreToSavepoint inline check
    missed. IO/permission failures now surface as HoodieException instead of
    being silently swallowed.
    
    The audit returns a tri-state result (PASSED / FAILED / INCONCLUSIVE) so
    transient cloud-storage timeouts are distinguishable from real
    audit failures; an audit_only mode lets users re-audit a previously
    completed restore by passing its restore_instant_time.
    
    Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
    
    * fix(spark): address PR review: public MDT guard, variable rename, test coverage
    
    Add a public deleteMetadataTableIfNecessaryBeforeRestore method to
    BaseHoodieWriteClient so callers that drive restore via restoreToInstant
    directly (e.g. the restore_to_instant procedure) can pre-check and
    pre-emptively delete the MDT before calling restoreToInstant. This
    closes the coverage gap identified in review: the procedure previously
    had no MDT protection after shouldDeleteMdtBeforeRestore was removed
    from restoreToInstant.
    
    The procedure now calls client.deleteMetadataTableIfNecessaryBeforeRestore
    before restoreToInstant and passes the returned boolean as
    initialMetadataTableIfNecessary, ensuring the penultimate/oldest
    compaction and timeline-start checks fire on the procedure path.
    
    Rename internal Scala variable restoreInstantTime -> startRestoreTimeArg
    for consistency with the start_restore_time parameter name.
    
    Strengthen testRestoreToInstantSkipsMdtCheckWhenMetadataDisabled: the
    test now verifies (a) deleteMetadataTableIfNecessaryBeforeRestore
    returns false and deletes the MDT for a target at/before the oldest
    compaction, and (b) restoreToInstant(target, false) then proceeds
    without invoking the guard.
    
    Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
    
    * fix(spark): address PR review round 2 β€” remove penultimate check, rename public API
    
    Remove the penultimate-compaction check from shouldDeleteMdtBeforeRestore.
    The check fired too aggressively: it deleted the MDT even when the MDT
    could restore successfully, and then the fresh re-bootstrap failed for
    record_index-enabled tables (testRLIWithMDTCleaning) or triggered an
    inline MDT compaction during bootstrap that broke the
    isMetadataTableRecreatedDuringRestore detection
    (testRestoreToSavepointDeletesMdtWhenTargetIsBeforePenultimateCompaction).
    Only the oldest-compaction and timeline-start checks remain, matching
    the original pre-PR behaviour.
    
    Add a catch clause for HoodieException (e.g. TableNotFoundException from
    a partially initialized MDT directory) so a corrupt-but-present MDT is
    treated as absent rather than hard-failing the restore.
    
    Rename deleteMetadataTableIfNecessaryBeforeRestore ->
    deleteMdtIfNecessaryBeforeRestore and flip the return value so that
    true = MDT was deleted (callers negate with ! when passing to
    restoreToInstant). Update RestoreToInstantProcedure and the
    corresponding Java test accordingly.
    
    Restore the @Deprecated annotation on the two-arg rollback overload that
    was accidentally dropped in a prior commit.
    
    Remove testRestoreToSavepointDeletesMdtWhenTargetIsBeforePenultimateCompaction
    which tested the now-removed penultimate check.
    
    Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
    
    ---------
    
    Co-authored-by: mahsoode <ma...@uber.com>
    Co-authored-by: Claude Opus 4.7 (1M context) <no...@anthropic.com>
    (cherry picked from commit e406e5d092d090926803946445ede3e6ac261af0)
---
 .../apache/hudi/client/BaseHoodieWriteClient.java  | 120 +++++---
 .../TestSavepointRestoreCopyOnWrite.java           |  89 ++++++
 .../hudi/command/procedures/HoodieProcedures.scala |   1 +
 .../procedures/RestoreToInstantProcedure.scala     | 309 +++++++++++++++++++++
 .../sql/hudi/procedure/TestRestoreProcedure.scala  | 258 +++++++++++++++++
 5 files changed, 739 insertions(+), 38 deletions(-)

diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
index df06df2fbbba..76ffe51a256d 100644
--- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
+++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java
@@ -92,6 +92,7 @@ import org.apache.hudi.metadata.HoodieTableMetadataUtil;
 import org.apache.hudi.metadata.HoodieTableMetadataWriter;
 import org.apache.hudi.metadata.MetadataPartitionType;
 import org.apache.hudi.metrics.HoodieMetrics;
+import org.apache.hudi.storage.StoragePath;
 import org.apache.hudi.table.BulkInsertPartitioner;
 import org.apache.hudi.table.HoodieTable;
 import org.apache.hudi.table.action.HoodieWriteMetadata;
@@ -846,44 +847,11 @@ public abstract class BaseHoodieWriteClient<T, I, K, O> extends BaseHoodieClient
    */
   public void restoreToSavepoint(String savepointTime) {
     boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
-    if (initializeMetadataTableIfNecessary) {
-      try {
-        // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
-        // or before the oldest compaction on MDT.
-        // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
-        HoodieTableMetaClient mdtMetaClient = HoodieTableMetaClient.builder()
-            .setConf(storageConf.newInstance())
-            .setBasePath(getMetadataTableBasePath(config.getBasePath())).build();
-        Option<HoodieInstant> oldestMdtCompaction = mdtMetaClient.getCommitTimeline().filterCompletedInstants().firstInstant();
-        boolean deleteMDT = false;
-        if (oldestMdtCompaction.isPresent()) {
-          if (LESSER_THAN_OR_EQUALS.test(savepointTime, oldestMdtCompaction.get().requestedTime())) {
-            log.warn("Deleting MDT during restore to {} as the savepoint is older than oldest compaction {} on MDT",
-                savepointTime, oldestMdtCompaction.get().requestedTime());
-            deleteMDT = true;
-          }
-        }
-
-        // The instant required to sync rollback to MDT has been archived and the mdt syncing will be failed
-        // So that we need to delete the whole MDT here.
-        if (!deleteMDT) {
-          HoodieInstant syncedInstant = mdtMetaClient.createNewInstant(HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, savepointTime);
-          if (mdtMetaClient.getCommitsTimeline().isBeforeTimelineStarts(syncedInstant.requestedTime())) {
-            log.warn("Deleting MDT during restore to {} as the savepoint is older than the MDT timeline {}",
-                savepointTime, mdtMetaClient.getCommitsTimeline().firstInstant().get().requestedTime());
-            deleteMDT = true;
-          }
-        }
-
-        if (deleteMDT) {
-          HoodieTableMetadataUtil.deleteMetadataTable(config.getBasePath(), context);
-          // rollbackToSavepoint action will try to bootstrap MDT at first but sync to MDT will fail at the current scenario.
-          // so that we need to disable metadata initialized here.
-          initializeMetadataTableIfNecessary = false;
-        }
-      } catch (Exception e) {
-        // Metadata directory does not exist
-      }
+    if (initializeMetadataTableIfNecessary && shouldDeleteMdtBeforeRestore(savepointTime)) {
+      HoodieTableMetadataUtil.deleteMetadataTable(config.getBasePath(), context);
+      // rollbackToSavepoint action will try to bootstrap MDT at first but sync to MDT will fail at the current scenario.
+      // so that we need to disable metadata initialized here.
+      initializeMetadataTableIfNecessary = false;
     }
 
     HoodieTable<T, I, K, O> table = initTable(WriteOperationType.UNKNOWN, Option.empty(), initializeMetadataTableIfNecessary);
@@ -894,6 +862,82 @@ public abstract class BaseHoodieWriteClient<T, I, K, O> extends BaseHoodieClient
     SavepointHelpers.validateSavepointRestore(table, savepointTime);
   }
 
+  /**
+   * Decides whether the metadata table (MDT) must be deleted before restoring the data table to
+   * {@code targetInstant}. Returns true when restoring would leave the MDT in an inconsistent
+   * state, specifically when any of the following holds:
+   * <ol>
+   *   <li>The target is at or before the oldest completed compaction. We cannot restore to before
+   *       the oldest compaction because we don't have base files before that time.</li>
+   *   <li>The target is before the MDT timeline start (the relevant history was archived away).</li>
+   * </ol>
+   * Returns false when the MDT directory does not exist or is not readable (nothing to delete or
+   * worry about). Wraps genuine IO failures ({@link IOException}) in a {@link HoodieException}
+   * so permission / network errors surface to the caller.
+   */
+  protected boolean shouldDeleteMdtBeforeRestore(String targetInstant) {
+    String mdtBasePath = getMetadataTableBasePath(config.getBasePath());
+    try {
+      // Cheap existence check first to avoid constructing an MDT meta client when there is no MDT.
+      if (!storage.exists(new StoragePath(mdtBasePath))) {
+        return false;
+      }
+      HoodieTableMetaClient mdtMetaClient = HoodieTableMetaClient.builder()
+          .setConf(storageConf.newInstance())
+          .setBasePath(mdtBasePath).build();
+      List<HoodieInstant> completedCompactions = mdtMetaClient.getCommitTimeline()
+          .filterCompletedInstants().getInstants();
+      Option<HoodieInstant> oldestMdtCompaction = completedCompactions.isEmpty()
+          ? Option.empty() : Option.of(completedCompactions.get(0));
+      if (oldestMdtCompaction.isPresent()
+          && LESSER_THAN_OR_EQUALS.test(targetInstant, oldestMdtCompaction.get().requestedTime())) {
+        log.warn("Deleting MDT before restore to {}: target is at or before oldest MDT compaction {}",
+            targetInstant, oldestMdtCompaction.get().requestedTime());
+        return true;
+      }
+      if (mdtMetaClient.getCommitsTimeline().isBeforeTimelineStarts(targetInstant)) {
+        log.warn("Deleting MDT before restore to {}: target is before MDT timeline start", targetInstant);
+        return true;
+      }
+      return false;
+    } catch (IOException e) {
+      throw new HoodieException(
+          "Failed to inspect MDT at " + mdtBasePath + " before restore to " + targetInstant
+              + " - refusing to silently proceed without an MDT integrity check.", e);
+    } catch (HoodieException e) {
+      // MDT directory exists but is not usable (e.g. TableNotFoundException from a partially
+      // initialized MDT). Treat as absent: no deletion needed, let the restore proceed.
+      log.warn("MDT at {} is present but could not be read ({}); skipping pre-check.",
+          mdtBasePath, e.getMessage());
+      return false;
+    }
+  }
+
+  /**
+   * Deletes the metadata table (MDT) if it would be left in an inconsistent state by a restore
+   * to {@code targetInstant}, and returns whether the MDT was actually deleted.
+   *
+   * <p>Callers that drive restore via {@link #restoreToInstant} directly (e.g. the
+   * {@code restore_to_instant} stored procedure) should call this method before invoking
+   * {@code restoreToInstant} and suppress MDT initialization when it returns {@code true}:
+   *
+   * <pre>{@code
+   * boolean mdtDeleted = client.deleteMdtIfNecessaryBeforeRestore(targetInstant);
+   * client.restoreToInstant(targetInstant, !mdtDeleted && enableMetadata);
+   * }</pre>
+   *
+   * @param targetInstant the instant the data table will be restored to
+   * @return {@code true} if the MDT was deleted (caller must not re-initialize it);
+   *         {@code false} otherwise (MDT either did not need deletion or does not exist)
+   */
+  public boolean deleteMdtIfNecessaryBeforeRestore(String targetInstant) {
+    if (shouldDeleteMdtBeforeRestore(targetInstant)) {
+      HoodieTableMetadataUtil.deleteMetadataTable(config.getBasePath(), context);
+      return true;
+    }
+    return false;
+  }
+
   @Deprecated
   public boolean rollback(final String commitInstantTime) throws HoodieRollbackException {
     HoodieTable<T, I, K, O> table = initTable(WriteOperationType.UNKNOWN, Option.empty());
diff --git a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestSavepointRestoreCopyOnWrite.java b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestSavepointRestoreCopyOnWrite.java
index 0be971630c34..5c3c83226073 100644
--- a/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestSavepointRestoreCopyOnWrite.java
+++ b/hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestSavepointRestoreCopyOnWrite.java
@@ -29,6 +29,8 @@ import org.apache.hudi.common.util.Option;
 import org.apache.hudi.config.HoodieWriteConfig;
 import org.apache.hudi.exception.HoodieException;
 import org.apache.hudi.metadata.HoodieTableMetadata;
+import org.apache.hudi.storage.HoodieStorageUtils;
+import org.apache.hudi.storage.StoragePath;
 import org.apache.hudi.table.HoodieSparkTable;
 import org.apache.hudi.testutils.HoodieClientTestBase;
 
@@ -41,6 +43,7 @@ import java.util.Objects;
 
 import static org.apache.hudi.common.testutils.HoodieTestUtils.INSTANT_GENERATOR;
 import static org.apache.hudi.metadata.HoodieTableMetadata.SOLO_COMMIT_TIMESTAMP;
+import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 
@@ -217,4 +220,90 @@ public class TestSavepointRestoreCopyOnWrite extends HoodieClientTestBase {
       assertRowNumberEqualsTo(20);
     }
   }
+
+  /**
+   * Two-assertion test for the MDT pre-check guard:
+   * <ol>
+   *   <li>{@code deleteMdtIfNecessaryBeforeRestore(firstCommit)} returns {@code true} (and
+   *       deletes the MDT) when the target is at or before the oldest MDT compaction,
+   *       confirming the guard logic fires for this table setup.</li>
+   *   <li>{@code restoreToInstant(firstCommit, false)} does NOT invoke the pre-check, so a
+   *       caller that explicitly opts out of MDT integration is never surprised by an implicit
+   *       MDT deletion.</li>
+   * </ol>
+   */
+  @Test
+  void testRestoreToInstantSkipsMdtCheckWhenMetadataDisabled() throws Exception {
+    HoodieWriteConfig hoodieWriteConfig = getConfigBuilder(HoodieFailedWritesCleaningPolicy.LAZY)
+        .withRollbackUsingMarkers(true)
+        .withMetadataConfig(HoodieMetadataConfig.newBuilder()
+            .enable(true)
+            .withMaxNumDeltaCommitsBeforeCompaction(4)
+            .build())
+        .build();
+    try (SparkRDDWriteClient client = getHoodieWriteClient(hoodieWriteConfig)) {
+      String firstCommit = null;
+      String prevInstant = HoodieTimeline.INIT_INSTANT_TS;
+      final int numRecords = 10;
+      // 5 inserts so the MDT goes through one compaction with maxDeltaCommits=4.
+      for (int i = 1; i <= 5; i++) {
+        String newCommitTime = WriteClientTestUtils.createNewInstantTime();
+        insertBatch(hoodieWriteConfig, client, newCommitTime, prevInstant, numRecords, SparkRDDWriteClient::insert,
+            false, true, numRecords, numRecords * i, 1, Option.empty(), INSTANT_GENERATOR);
+        prevInstant = newCommitTime;
+        if (i == 1) {
+          firstCommit = newCommitTime;
+        }
+      }
+      assertRowNumberEqualsTo(50);
+
+      String mdtBasePath = HoodieTableMetadata.getMetadataTableBasePath(hoodieWriteConfig.getBasePath());
+      assertTrue(HoodieStorageUtils.getStorage(mdtBasePath, storageConf).exists(new StoragePath(mdtBasePath)),
+          "MDT directory should exist before any pre-check");
+
+      // Assertion 1: deleteMdtIfNecessaryBeforeRestore detects that firstCommit is at or before
+      // the oldest MDT compaction, deletes the MDT, and returns true.
+      boolean mdtDeleted = client.deleteMdtIfNecessaryBeforeRestore(
+          Objects.requireNonNull(firstCommit, "first commit should not be null"));
+      assertTrue(mdtDeleted,
+          "deleteMdtIfNecessaryBeforeRestore should return true when target is at or before the oldest MDT compaction");
+      assertFalse(HoodieStorageUtils.getStorage(mdtBasePath, storageConf).exists(new StoragePath(mdtBasePath)),
+          "MDT directory should have been deleted by deleteMdtIfNecessaryBeforeRestore");
+
+      // Assertion 2: restoreToInstant with initialMetadataTableIfNecessary=false does NOT invoke
+      // the pre-check β€” the MDT (now absent) is not touched. The restore proceeds without MDT.
+      client.restoreToInstant(firstCommit, false);
+      assertRowNumberEqualsTo(numRecords);
+    }
+  }
+
+  /**
+   * Regression coverage for the {@code restoreToSavepoint} refactor. After replacing the inline
+   * MDT pre-check with a call to the centralized helper, the common case (target after the
+   * oldest MDT compaction, no MDT delete needed) must still work end-to-end.
+   */
+  @Test
+  void testRestoreToSavepointStillWorksAfterRefactor() throws Exception {
+    HoodieWriteConfig hoodieWriteConfig = getConfigBuilder(HoodieFailedWritesCleaningPolicy.LAZY)
+        .withRollbackUsingMarkers(true)
+        .build();
+    try (SparkRDDWriteClient client = getHoodieWriteClient(hoodieWriteConfig)) {
+      String savepointCommit = null;
+      String prevInstant = HoodieTimeline.INIT_INSTANT_TS;
+      final int numRecords = 10;
+      for (int i = 1; i <= 4; i++) {
+        String newCommitTime = WriteClientTestUtils.createNewInstantTime();
+        insertBatch(hoodieWriteConfig, client, newCommitTime, prevInstant, numRecords, SparkRDDWriteClient::insert,
+            false, true, numRecords, numRecords * i, 1, Option.empty(), INSTANT_GENERATOR);
+        prevInstant = newCommitTime;
+        if (i == 2) {
+          savepointCommit = newCommitTime;
+          client.savepoint("user1", "Savepoint for 2nd commit");
+        }
+      }
+      assertRowNumberEqualsTo(40);
+      client.restoreToSavepoint(Objects.requireNonNull(savepointCommit, "restore commit should not be null"));
+      assertRowNumberEqualsTo(20);
+    }
+  }
 }
diff --git a/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/HoodieProcedures.scala b/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/HoodieProcedures.scala
index 4319ce77c1c2..027c9f8c2882 100644
--- a/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/HoodieProcedures.scala
+++ b/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/HoodieProcedures.scala
@@ -39,6 +39,7 @@ object HoodieProcedures {
       ,(DeleteSavepointProcedure.NAME, DeleteSavepointProcedure.builder)
       ,(RollbackToSavepointProcedure.NAME, RollbackToSavepointProcedure.builder)
       ,(RollbackToInstantTimeProcedure.NAME, RollbackToInstantTimeProcedure.builder)
+      ,(RestoreToInstantProcedure.NAME, RestoreToInstantProcedure.builder)
       ,(RunClusteringProcedure.NAME, RunClusteringProcedure.builder)
       ,(ShowClusteringProcedure.NAME, ShowClusteringProcedure.builder)
       ,(ShowCommitsProcedure.NAME, ShowCommitsProcedure.builder)
diff --git a/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RestoreToInstantProcedure.scala b/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RestoreToInstantProcedure.scala
new file mode 100644
index 000000000000..ca5a2fc924ae
--- /dev/null
+++ b/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/RestoreToInstantProcedure.scala
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.HoodieCLIUtils
+import org.apache.hudi.avro.model.HoodieRestoreMetadata
+import org.apache.hudi.client.SparkRDDWriteClient
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.fs.ConsistencyGuardConfig
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.HoodieInstant
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.exception.HoodieException
+import org.apache.hudi.hadoop.fs.HadoopFSUtils
+import org.apache.hudi.storage.StoragePath
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.hudi.command.procedures.RestoreToInstantProcedure._
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Stored procedure to perform a full point-in-time table restore to a given instant.
+ *
+ * Unlike [[RollbackToSavepointProcedure]] (which requires a savepoint at the target instant),
+ * this procedure calls restoreToInstant() directly and works on any arbitrary instant on the
+ * active timeline.
+ *
+ * Parameters:
+ *   - table / path: identifies the Hudi table (one must be provided)
+ *   - instant_time: target commit to restore to (required when audit_only=false; must be omitted
+ *                   when audit_only=true)
+ *   - start_restore_time: the restore operation's own timeline timestamp (the start_restore_time
+ *                   value returned by a prior restore_to_instant call). Required when
+ *                   audit_only=true; must be omitted otherwise.
+ *   - enable_metadata: whether the metadata table is enabled (default: true)
+ *   - rollback_parallelism: Spark parallelism for rollback and audit operations (default: 4)
+ *   - enable_consistency_guard: enable consistency guard for file existence checks (default: false)
+ *   - audit_post_restore: after restoring, verify that all successfully-deleted files are absent (default: false)
+ *   - audit_only: skip the restore and only audit a previously completed restore instant (default: false)
+ *
+ * Output columns:
+ *   - restore_result: true if restore succeeded; null if audit_only=true
+ *   - start_restore_time: the restore operation's own timeline timestamp; null if audit_only=true.
+ *                         Pass this value as start_restore_time to re-run the audit later.
+ *   - time_taken_in_millis: restore duration; null if audit_only=true
+ *   - instants_rolled_back: number of commits rolled back; null if audit_only=true
+ *   - audit_result: one of "PASSED" / "FAILED" / "INCONCLUSIVE" when an audit ran; null otherwise.
+ *                   INCONCLUSIVE means at least one file existence check threw an IOException
+ *                   (e.g. transient cloud-storage timeout) β€” re-run audit_only=true to retry.
+ */
+class RestoreToInstantProcedure extends BaseProcedure with ProcedureBuilder with Logging {
+
+  private val PARAMETERS = Array[ProcedureParameter](
+    ProcedureParameter.optional(0, "table", DataTypes.StringType),
+    ProcedureParameter.optional(1, "instant_time", DataTypes.StringType),
+    ProcedureParameter.optional(2, "enable_metadata", DataTypes.BooleanType, true),
+    ProcedureParameter.optional(3, "rollback_parallelism", DataTypes.IntegerType, 4),
+    ProcedureParameter.optional(4, "enable_consistency_guard", DataTypes.BooleanType, false),
+    ProcedureParameter.optional(5, "audit_post_restore", DataTypes.BooleanType, false),
+    ProcedureParameter.optional(6, "audit_only", DataTypes.BooleanType, false),
+    ProcedureParameter.optional(7, "path", DataTypes.StringType),
+    ProcedureParameter.optional(8, "start_restore_time", DataTypes.StringType)
+  )
+
+  private val OUTPUT_TYPE = new StructType(Array[StructField](
+    StructField("restore_result", DataTypes.BooleanType, nullable = true, Metadata.empty),
+    StructField("start_restore_time", DataTypes.StringType, nullable = true, Metadata.empty),
+    StructField("time_taken_in_millis", DataTypes.LongType, nullable = true, Metadata.empty),
+    StructField("instants_rolled_back", DataTypes.LongType, nullable = true, Metadata.empty),
+    StructField("audit_result", DataTypes.StringType, nullable = true, Metadata.empty)
+  ))
+
+  def parameters: Array[ProcedureParameter] = PARAMETERS
+
+  def outputType: StructType = OUTPUT_TYPE
+
+  override def call(args: ProcedureArgs): Seq[Row] = {
+    super.checkArgs(PARAMETERS, args)
+
+    val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+    val instantTime = getArgValueOrDefault(args, PARAMETERS(1))
+    val enableMetadata = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Boolean]
+    val rollbackParallelism = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Int]
+    val enableConsistencyGuard = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[Boolean]
+    val shouldAuditPostRestore = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[Boolean]
+    val auditOnly = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[Boolean]
+    val tablePath = getArgValueOrDefault(args, PARAMETERS(7))
+    val startRestoreTimeArg = getArgValueOrDefault(args, PARAMETERS(8))
+
+    // Cross-validation: each of (instant_time, start_restore_time) has one unambiguous meaning.
+    if (!auditOnly && instantTime.isEmpty) {
+      throw new HoodieException("instant_time is required when audit_only=false.")
+    }
+    if (auditOnly && startRestoreTimeArg.isEmpty) {
+      throw new HoodieException(
+        "start_restore_time is required when audit_only=true. " +
+          "Pass the start_restore_time value from a prior restore_to_instant call.")
+    }
+    if (!auditOnly && startRestoreTimeArg.isDefined) {
+      throw new HoodieException("start_restore_time may only be specified when audit_only=true.")
+    }
+    if (auditOnly && instantTime.isDefined) {
+      throw new HoodieException(
+        "instant_time may only be specified when audit_only=false. " +
+          "Use start_restore_time to identify a previously executed restore.")
+    }
+    if (auditOnly && shouldAuditPostRestore) {
+      logWarning("Both audit_only and audit_post_restore are set. Only audit_only will be honored.")
+    }
+
+    val basePath = getBasePath(tableName, tablePath)
+
+    val confs = Map(
+      HoodieMetadataConfig.ENABLE.key() -> enableMetadata.toString,
+      HoodieWriteConfig.ROLLBACK_PARALLELISM_VALUE.key() -> rollbackParallelism.toString,
+      HoodieWriteConfig.ROLLBACK_USING_MARKERS_ENABLE.key() -> "false"
+    ) ++ (if (enableConsistencyGuard) Map(ConsistencyGuardConfig.ENABLE.key() -> "true") else Map.empty)
+
+    val metaClient = createMetaClient(jsc, basePath)
+
+    // Nullable boxed types so Row can hold null for audit_only runs
+    var restoreResult: java.lang.Boolean = null
+    var startRestoreTime: String = null
+    var timeTakenInMillis: java.lang.Long = null
+    var instantsRolledBack: java.lang.Long = null
+
+    if (!auditOnly) {
+      val targetInstant = instantTime.get.asInstanceOf[String]
+      var client: SparkRDDWriteClient[_] = null
+      try {
+        client = HoodieCLIUtils.createHoodieWriteClient(sparkSession, basePath, confs,
+          tableName.asInstanceOf[Option[String]])
+        // Pre-check: if the target instant is at or before the oldest MDT compaction or before the
+        // MDT timeline start, pre-emptively delete the MDT so restoreToInstant does not leave it
+        // inconsistent. deleteMdtIfNecessaryBeforeRestore returns true when the MDT was deleted
+        // (caller must not re-initialize it), false otherwise.
+        val mdtDeleted = if (enableMetadata) {
+          client.deleteMdtIfNecessaryBeforeRestore(targetInstant)
+        } else false
+        val restoreMetadata = client.restoreToInstant(targetInstant, !mdtDeleted && enableMetadata)
+        restoreResult = true
+        startRestoreTime = restoreMetadata.getStartRestoreTime
+        timeTakenInMillis = restoreMetadata.getTimeTakenInMillis
+        instantsRolledBack = restoreMetadata.getInstantsToRollback.size().toLong
+      } finally {
+        if (client != null) {
+          client.close()
+        }
+      }
+      if (tableName.isDefined) {
+        spark.catalog.refreshTable(tableName.get.asInstanceOf[String])
+      }
+      // getActiveTimeline is lazily cached; reload so the new .restore instant is visible to the audit.
+      if (shouldAuditPostRestore) {
+        metaClient.reloadActiveTimeline()
+      }
+    }
+
+    var auditResult: String = null
+    if (auditOnly || shouldAuditPostRestore) {
+      val restoreInstant: HoodieInstant = if (auditOnly) {
+        val ts = startRestoreTimeArg.get.asInstanceOf[String]
+        val instants = metaClient.getActiveTimeline.getRestoreTimeline.filterCompletedInstants
+          .getInstants.asScala
+        instants.find(_.requestedTime().equals(ts)).getOrElse(
+          throw new HoodieException(s"No completed restore instant found for $ts. " +
+            "Pass the start_restore_time from a prior restore_to_instant call as start_restore_time.")
+        )
+      } else {
+        // Use startRestoreTime captured from the restore metadata β€” more deterministic than
+        // lastInstant() since another concurrent restore could otherwise land in between.
+        val ts = startRestoreTime
+        val instants = metaClient.getActiveTimeline.getRestoreTimeline.filterCompletedInstants
+          .getInstants.asScala
+        instants.find(_.requestedTime().equals(ts)).getOrElse(
+          throw new HoodieException(s"No completed restore instant found for $ts after restore.")
+        )
+      }
+      auditResult = auditPostRestore(metaClient, basePath, restoreInstant, rollbackParallelism)
+    }
+
+    Seq(Row(restoreResult, startRestoreTime, timeTakenInMillis, instantsRolledBack, auditResult))
+  }
+
+  /**
+   * Verifies that all files expected to have been deleted by a restore operation are actually
+   * absent from storage. Returns one of "PASSED", "FAILED", or "INCONCLUSIVE":
+   *   - PASSED: every file expected to be absent is in fact absent.
+   *   - FAILED: at least one file is still present after restore.
+   *   - INCONCLUSIVE: no file was confirmed present, but at least one existence check threw
+   *                   an IOException (e.g. transient cloud-storage timeout). Re-run with
+   *                   audit_only=true to retry.
+   */
+  private def auditPostRestore(
+      metaClient: HoodieTableMetaClient,
+      basePath: String,
+      restoreInstant: HoodieInstant,
+      rollbackParallelism: Int): String = {
+    try {
+      val restoreMetadata: HoodieRestoreMetadata =
+        metaClient.getActiveTimeline.readRestoreMetadata(restoreInstant)
+
+      val filesToCheck = new java.util.ArrayList[String]()
+      restoreMetadata.getHoodieRestoreMetadata.values.asScala.foreach { rollbackList =>
+        rollbackList.asScala.foreach { rollback =>
+          rollback.getPartitionMetadata.asScala.foreach { case (partition, pm) =>
+            val partitionPathStr = basePath + Path.SEPARATOR + partition
+            val partitionStoragePath = new StoragePath(partitionPathStr)
+            if (!metaClient.getStorage.exists(partitionStoragePath)) {
+              logInfo(s"Partition path $partitionStoragePath does not exist. Skipping audit for its files.")
+            } else {
+              // Use .toString() to preserve the full absolute path. Using .getName() would strip
+              // the path to just the filename, making the subsequent FS.exists() check vacuously pass.
+              // Only check getSuccessDeleteFiles: these are the files the restore intended to delete
+              // and whose absence we can meaningfully verify. Files in getFailedDeleteFiles already
+              // failed to be deleted and are expected to still be present.
+              pm.getSuccessDeleteFiles.asScala.foreach(f =>
+                filesToCheck.add(new Path(partitionPathStr, f).toString))
+            }
+          }
+        }
+      }
+
+      if (filesToCheck.isEmpty) {
+        logInfo(s"No files to audit for restore instant ${restoreInstant.requestedTime()}")
+        "PASSED"
+      } else {
+        // HadoopFSUtils.getStorageConfWithCopy returns a Serializable StorageConfiguration; the
+        // closure captures only storageConf, so Spark's ClosureCleaner can serialize it cleanly.
+        // HadoopFSUtils.getFs internally calls prepareHadoopConf, preserving HOODIE_ENV_* (e.g.
+        // S3A) injection at cloud DCs.
+        val storageConf = HadoopFSUtils.getStorageConfWithCopy(jsc.hadoopConfiguration())
+        val outcomes = jsc.parallelize(filesToCheck, Math.max(1, rollbackParallelism))
+          .map { pathStr =>
+            val p = new Path(pathStr)
+            try {
+              val exists = HadoopFSUtils.getFs(p, storageConf.unwrap()).exists(p)
+              val status: AuditFileStatus = if (exists) Present else Absent
+              (pathStr, status)
+            } catch {
+              case e: Exception => (pathStr, IndeterminateError(e.getMessage))
+            }
+          }
+          .collect()
+          .asScala
+          .toArray
+
+        val present = outcomes.collect { case (p, Present) => p }
+        val indeterminate = outcomes.collect { case (p, IndeterminateError(msg)) => (p, msg) }
+        if (present.nonEmpty) {
+          logError(s"Restore audit FAILED for instant ${restoreInstant.requestedTime()}: " +
+            s"${present.length} file(s) still present, e.g. ${present.take(5).mkString(", ")}")
+          "FAILED"
+        } else if (indeterminate.nonEmpty) {
+          logWarning(s"Restore audit INCONCLUSIVE for instant ${restoreInstant.requestedTime()}: " +
+            s"${indeterminate.length} file(s) had IO errors, e.g. ${indeterminate.take(5).mkString(", ")}")
+          "INCONCLUSIVE"
+        } else {
+          logInfo(s"Restore audit PASSED for instant ${restoreInstant.requestedTime()}: " +
+            s"${outcomes.length} file(s) confirmed absent.")
+          "PASSED"
+        }
+      }
+    } catch {
+      case e: Exception =>
+        logError(s"Exception during restore audit for instant ${restoreInstant.requestedTime()}", e)
+        "INCONCLUSIVE"
+    }
+  }
+
+  override def build: Procedure = new RestoreToInstantProcedure()
+}
+
+object RestoreToInstantProcedure {
+  val NAME: String = "restore_to_instant"
+
+  def builder: Supplier[ProcedureBuilder] = new Supplier[ProcedureBuilder] {
+    override def get(): RestoreToInstantProcedure = new RestoreToInstantProcedure()
+  }
+
+  // ADT for per-file audit outcomes. Defined at object level (NOT inside auditPostRestore) so the
+  // generated bytecode does not embed a synthetic outer-class reference; otherwise Spark's
+  // ClosureCleaner cannot serialize the closure that returns these values from executors.
+  private sealed trait AuditFileStatus extends Serializable
+  private case object Absent extends AuditFileStatus
+  private case object Present extends AuditFileStatus
+  private case class IndeterminateError(message: String) extends AuditFileStatus
+}
diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestRestoreProcedure.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestRestoreProcedure.scala
new file mode 100644
index 000000000000..7674fa6abaa8
--- /dev/null
+++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestRestoreProcedure.scala
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.procedure
+
+class TestRestoreProcedure extends HoodieSparkProcedureTestBase {
+
+  private def createTableAndInsertData(tableName: String, tablePath: String, tableType: String = "cow"): Array[String] = {
+    spark.sql(
+      s"""
+         |create table $tableName (
+         |  id int,
+         |  name string,
+         |  price double,
+         |  ts long
+         |) using hudi
+         | location '$tablePath'
+         | tblproperties (
+         |  primaryKey = 'id',
+         |  preCombineField = 'ts',
+         |  type = '$tableType'
+         | )
+       """.stripMargin)
+    spark.sql(s"insert into $tableName select 1, 'a1', 10.0, 1000")
+    spark.sql(s"insert into $tableName select 2, 'a2', 20.0, 1500")
+    spark.sql(s"insert into $tableName select 3, 'a3', 30.0, 2000")
+    spark.sql(s"insert into $tableName select 4, 'a4', 40.0, 2500")
+    spark.sql(s"call show_commits(table => '$tableName')").collect()
+      .map(_.getString(0)).sorted
+  }
+
+  test("Test restore_to_instant basic CoW") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      val commits = createTableAndInsertData(tableName, tablePath)
+      assertResult(4)(commits.length)
+
+      // restore to after the 2nd commit β€” only commits(0) and commits(1) should remain
+      val result = spark.sql(
+        s"call restore_to_instant(table => '$tableName', instant_time => '${commits(1)}')"
+      ).collect()
+
+      assertResult(1)(result.length)
+      assertResult(true)(result(0).getBoolean(0))     // restore_result
+      assert(result(0).getString(1) != null)          // start_restore_time (dynamic timestamp)
+      assert(result(0).getLong(2) >= 0)               // time_taken_in_millis
+      assert(result(0).getLong(3) >= 0L)              // instants_rolled_back
+      assertResult(true)(result(0).isNullAt(4))       // audit_result = null (audit not requested)
+
+      // verify data reverted to 2 records
+      val count = spark.sql(s"select count(*) from $tableName").collect()(0).getLong(0)
+      assertResult(2)(count)
+    }
+  }
+
+  test("Test restore_to_instant basic MoR") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      val commits = createTableAndInsertData(tableName, tablePath, tableType = "mor")
+      assertResult(4)(commits.length)
+
+      val result = spark.sql(
+        s"call restore_to_instant(table => '$tableName', instant_time => '${commits(1)}')"
+      ).collect()
+
+      assertResult(1)(result.length)
+      assertResult(true)(result(0).getBoolean(0))
+      assert(result(0).getString(1) != null)
+      assert(result(0).getLong(2) >= 0)
+      assert(result(0).getLong(3) >= 0L)
+      assertResult(true)(result(0).isNullAt(4))
+
+      val count = spark.sql(s"select count(*) from $tableName").collect()(0).getLong(0)
+      assertResult(2)(count)
+    }
+  }
+
+  test("Test restore_to_instant using path parameter") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      val commits = createTableAndInsertData(tableName, tablePath)
+      assertResult(4)(commits.length)
+
+      val result = spark.sql(
+        s"call restore_to_instant(path => '$tablePath', instant_time => '${commits(1)}')"
+      ).collect()
+
+      assertResult(1)(result.length)
+      assertResult(true)(result(0).getBoolean(0))
+      assert(result(0).isNullAt(4))
+
+      val count = spark.sql(s"select count(*) from $tableName").collect()(0).getLong(0)
+      assertResult(2)(count)
+    }
+  }
+
+  test("Test restore_to_instant with audit_post_restore") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      val commits = createTableAndInsertData(tableName, tablePath)
+      assertResult(4)(commits.length)
+
+      val result = spark.sql(
+        s"""call restore_to_instant(
+           |  table => '$tableName',
+           |  instant_time => '${commits(1)}',
+           |  audit_post_restore => true
+           |)""".stripMargin
+      ).collect()
+
+      assertResult(1)(result.length)
+      assertResult(true)(result(0).getBoolean(0))     // restore_result
+      assert(result(0).getString(1) != null)          // start_restore_time
+      // audit_result should be "PASSED": all rolled-back files are absent
+      assertResult(false)(result(0).isNullAt(4))
+      assertResult("PASSED")(result(0).getString(4))
+
+      val count = spark.sql(s"select count(*) from $tableName").collect()(0).getLong(0)
+      assertResult(2)(count)
+    }
+  }
+
+  test("Test restore_to_instant audit_only mode") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      val commits = createTableAndInsertData(tableName, tablePath)
+      assertResult(4)(commits.length)
+
+      // Round 1: perform the restore and capture the restore operation's own timeline timestamp
+      val restoreRows = spark.sql(
+        s"call restore_to_instant(table => '$tableName', instant_time => '${commits(1)}')"
+      ).collect()
+      assertResult(true)(restoreRows(0).getBoolean(0))
+      // start_restore_time is the restore instant's timeline timestamp β€” NOT commits(1)
+      val restoreInstantTs = restoreRows(0).getString(1)
+      assert(restoreInstantTs != null)
+      assert(restoreInstantTs != commits(1))
+
+      // Round 2: audit_only using start_restore_time (the original target commit is not needed)
+      val auditRows = spark.sql(
+        s"""call restore_to_instant(
+           |  table => '$tableName',
+           |  audit_only => true,
+           |  start_restore_time => '$restoreInstantTs'
+           |)""".stripMargin
+      ).collect()
+
+      assertResult(1)(auditRows.length)
+      assertResult(true)(auditRows(0).isNullAt(0))    // restore_result = null (no restore performed)
+      assertResult(true)(auditRows(0).isNullAt(1))    // start_restore_time = null
+      assertResult(true)(auditRows(0).isNullAt(2))    // time_taken_in_millis = null
+      assertResult(true)(auditRows(0).isNullAt(3))    // instants_rolled_back = null
+      assertResult(false)(auditRows(0).isNullAt(4))   // audit_result is present
+      assertResult("PASSED")(auditRows(0).getString(4))
+    }
+  }
+
+  test("Test restore_to_instant audit_only with non-existent restore instant throws") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      createTableAndInsertData(tableName, tablePath)
+
+      // start_restore_time pointing at a timestamp that has never been a restore instant.
+      assertThrows[Exception] {
+        spark.sql(
+          s"""call restore_to_instant(
+             |  table => '$tableName',
+             |  audit_only => true,
+             |  start_restore_time => '19700101000000000'
+             |)""".stripMargin
+        ).collect()
+      }
+    }
+  }
+
+  test("Test restore_to_instant cross-validation: audit_only=true requires start_restore_time") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      createTableAndInsertData(tableName, tablePath)
+
+      assertThrows[Exception] {
+        spark.sql(
+          s"call restore_to_instant(table => '$tableName', audit_only => true)"
+        ).collect()
+      }
+    }
+  }
+
+  test("Test restore_to_instant cross-validation: audit_only=false rejects start_restore_time") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      val commits = createTableAndInsertData(tableName, tablePath)
+
+      assertThrows[Exception] {
+        spark.sql(
+          s"""call restore_to_instant(
+             |  table => '$tableName',
+             |  instant_time => '${commits(1)}',
+             |  start_restore_time => '20990101000000000'
+             |)""".stripMargin
+        ).collect()
+      }
+    }
+  }
+
+  test("Test restore_to_instant cross-validation: audit_only=true rejects instant_time") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      val commits = createTableAndInsertData(tableName, tablePath)
+
+      assertThrows[Exception] {
+        spark.sql(
+          s"""call restore_to_instant(
+             |  table => '$tableName',
+             |  audit_only => true,
+             |  instant_time => '${commits(1)}',
+             |  start_restore_time => '20990101000000000'
+             |)""".stripMargin
+        ).collect()
+      }
+    }
+  }
+
+  test("Test restore_to_instant cross-validation: audit_only=false requires instant_time") {
+    withTempDir { tmp =>
+      val tableName = generateTableName
+      val tablePath = tmp.getCanonicalPath + "/" + tableName
+      createTableAndInsertData(tableName, tablePath)
+
+      assertThrows[Exception] {
+        spark.sql(s"call restore_to_instant(table => '$tableName')").collect()
+      }
+    }
+  }
+}


(hudi) 09/16: perf: Reduce unnecessary FSDataOutputStream#hsync to enhance append performance (#17517)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit b85f1cc3e14a3ff89f843b784d4346a44e5e69c2
Author: chaoyang <ch...@apache.org>
AuthorDate: Sun May 10 11:03:19 2026 +0800

    perf: Reduce unnecessary FSDataOutputStream#hsync to enhance append performance (#17517)
    
    * perf: Reduce unnecessary `FSDataOutputStream#hsync` to enhance append performance
    
    1. Reduce unnecessary `FSDataOutputStream#hsync` to enhance append performance
    
    Signed-off-by: TheR1sing3un <ch...@apache.org>
    
    * feat: flush behavior compatible with the block append mode
    
    1. flush behavior compatible with the block append mode
    
    Signed-off-by: TheR1sing3un <ch...@apache.org>
    
    * fixup: address review - drop syncDuringFlush, expose explicit sync()
    
    Following @danny0405's suggestion in the PR review, ensure only
    commit-level visibility on the production path:
    
    - Remove the `withSyncDuringFlush` builder option and the
      `flush(boolean)` overload on HoodieLogFormatWriter; the production
      path no longer flushes or hsyncs at appendBlocks.
    - Expose `Writer#sync()` (flush + hsync) as an explicit API for tests
      that assert per-append visibility on the underlying file system.
    - closeStream still calls sync() once before close so a closed writer
      guarantees data is persisted to DataNodes.
    - Update tests that previously relied on `withSyncDuringFlush(true)`
      to call `writer.sync()` explicitly before per-append FileStatus
      size assertions, and rename the related assertion message to drop
      the misleading "auto-flushed" wording.
    
    ---------
    
    Signed-off-by: TheR1sing3un <ch...@apache.org>
    (cherry picked from commit 1fffd70db1cb7b7e0441a09842b0d152dfaafa2e)
---
 .../apache/hudi/common/table/log/HoodieLogFormat.java   | 11 +++++++++++
 .../hudi/common/table/log/HoodieLogFormatWriter.java    | 17 +++++++++++------
 .../hudi/common/functional/TestHoodieLogFormat.java     |  9 ++++++---
 3 files changed, 28 insertions(+), 9 deletions(-)

diff --git a/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormat.java b/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormat.java
index a1103705c3c2..2c536ec07722 100644
--- a/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormat.java
+++ b/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormat.java
@@ -85,6 +85,17 @@ public interface HoodieLogFormat {
     AppendResult appendBlocks(List<HoodieLogBlock> blocks) throws IOException, InterruptedException;
 
     long getCurrentSize() throws IOException;
+
+    /**
+     * Force previously appended blocks to durable storage so that downstream
+     * readers can observe them before this writer is closed.
+     *
+     * <p>Production code paths typically rely on {@link #close()} for
+     * commit-level visibility and do not need to call this. It is exposed
+     * mainly for tests that assert per-append visibility on the underlying
+     * file system.
+     */
+    void sync() throws IOException;
   }
 
   /**
diff --git a/hudi-hadoop-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java b/hudi-hadoop-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java
index 23a69699a43e..1d6acc8d85ef 100644
--- a/hudi-hadoop-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java
+++ b/hudi-hadoop-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java
@@ -179,8 +179,10 @@ public class HoodieLogFormatWriter implements HoodieLogFormat.Writer {
       }
       sizeWritten +=  outputStream.size() - startSize;
     }
-    // Flush all blocks to disk
-    flush();
+    // No flush/hsync here: append-time visibility is not part of the contract.
+    // Downstream readers only need commit-level visibility, which is provided
+    // when the writer is closed (see closeStream) or when callers explicitly
+    // invoke sync().
 
     AppendResult result = new AppendResult(logFile, startPos, sizeWritten);
     // roll over if size is past the threshold
@@ -236,20 +238,23 @@ public class HoodieLogFormatWriter implements HoodieLogFormat.Writer {
 
   private void closeStream() throws IOException {
     if (output != null) {
-      flush();
+      // Persist all buffered data to DataNodes before closing so downstream
+      // readers can observe a fully-written log file at commit-level visibility.
+      sync();
       output.close();
       output = null;
       closed = true;
     }
   }
 
-  private void flush() throws IOException {
+  @Override
+  public void sync() throws IOException {
     if (output == null) {
       return; // Presume closed
     }
     output.flush();
-    // NOTE : the following API call makes sure that the data is flushed to disk on DataNodes (akin to POSIX fsync())
-    // See more details here : https://issues.apache.org/jira/browse/HDFS-744
+    // NOTE: the following API call makes sure that the data is flushed to disk on DataNodes (akin to POSIX fsync())
+    // See more details here: https://issues.apache.org/jira/browse/HDFS-744
     output.hsync();
   }
 
diff --git a/hudi-hadoop-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormat.java b/hudi-hadoop-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormat.java
index cde861f8c5f2..7e3ab4ebbbb8 100755
--- a/hudi-hadoop-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormat.java
+++ b/hudi-hadoop-common/src/test/java/org/apache/hudi/common/functional/TestHoodieLogFormat.java
@@ -234,8 +234,9 @@ public class TestHoodieLogFormat extends HoodieCommonTestHarness {
 
     long size = writer.getCurrentSize();
     assertTrue(size > 0, "We just wrote a block - size should be > 0");
+    writer.sync();
     assertEquals(size, storage.getPathInfo(writer.getLogFile().getPath()).getLength(),
-        "Write should be auto-flushed. The size reported by FileStatus and the writer should match");
+        "After explicit sync, FileStatus length should match the writer's reported size");
     assertEquals(size, result.size());
     assertEquals(writer.getLogFile(), result.logFile());
     assertEquals(0, result.offset());
@@ -376,8 +377,9 @@ public class TestHoodieLogFormat extends HoodieCommonTestHarness {
     writer.appendBlock(dataBlock);
     long size2 = writer.getCurrentSize();
     assertTrue(size2 > size1, "We just wrote a new block - size2 should be > size1");
+    writer.sync();
     assertEquals(size2, storage.getPathInfo(writer.getLogFile().getPath()).getLength(),
-        "Write should be auto-flushed. The size reported by FileStatus and the writer should match");
+        "After explicit sync, FileStatus length should match the writer's reported size");
     writer.close();
 
     // Close and Open again and append 100 more records
@@ -394,8 +396,9 @@ public class TestHoodieLogFormat extends HoodieCommonTestHarness {
     writer.appendBlock(dataBlock);
     long size3 = writer.getCurrentSize();
     assertTrue(size3 > size2, "We just wrote a new block - size3 should be > size2");
+    writer.sync();
     assertEquals(size3, storage.getPathInfo(writer.getLogFile().getPath()).getLength(),
-        "Write should be auto-flushed. The size reported by FileStatus and the writer should match");
+        "After explicit sync, FileStatus length should match the writer's reported size");
     writer.close();
 
     // Cannot get the current size after closing the log


(hudi) 02/16: test(spark): Add date logical type test to TestAvroConversionUtils (#18584)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit d030eb17dd899d7f59d9f7d9a5c2e8cb1e633193
Author: ashokkumar-allu <65...@users.noreply.github.com>
AuthorDate: Fri May 1 12:32:02 2026 -0500

    test(spark): Add date logical type test to TestAvroConversionUtils (#18584)
    
    * [MINOR] Add date logical type test to TestAvroConversionUtils
    
    Add a test case that verifies createConverterToRow correctly handles
    Avro's date logical type (int with logicalType=date), ensuring the
    conversion from epoch-days integer to java.sql.Date preserves the
    correct date value.
    
    * [MINOR] Address review comments: rename convertor to converter, add comments
    
    ---------
    
    Co-authored-by: gallu <ga...@uber.com>
    (cherry picked from commit f7508ded95a534f5e6e5ffc5150a83a5662d05db)
---
 .../org/apache/hudi/TestAvroConversionUtils.scala  | 27 ++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestAvroConversionUtils.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestAvroConversionUtils.scala
index 5f4ec78c13cf..0430dd081237 100644
--- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestAvroConversionUtils.scala
+++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestAvroConversionUtils.scala
@@ -28,6 +28,7 @@ import org.apache.spark.sql.types.{ArrayType, BinaryType, DataType, DataTypes, M
 import org.scalatest.{FunSuite, Matchers}
 
 import java.nio.ByteBuffer
+import java.time.LocalDate
 import java.util.Objects
 
 class TestAvroConversionUtils extends FunSuite with Matchers {
@@ -456,4 +457,30 @@ class TestAvroConversionUtils extends FunSuite with Matchers {
       HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(struct, "SchemaName", "SchemaNS")
     } should have message "Duplicate field name in record SchemaNS.SchemaName: name type:UNION pos:2 and name type:UNION pos:1."
   }
+
+  test("Logical type: date") {
+    val dateSchema = s"""
+      {
+        "namespace": "logical",
+        "type": "record",
+        "name": "test",
+        "fields": [
+          {"name": "date", "type": {"type": "int", "logicalType": "date"}}
+        ]
+      }
+    """
+
+    val dateInputData = Seq(7, 365, 0) // one week, one year, epoch
+    val schema = HoodieSchema.parse(dateSchema)
+    val converter = AvroConversionUtils.createConverterToRow(schema, HoodieSchemaConversionUtils.convertHoodieSchemaToStructType(schema))
+
+    val dateOutputData = dateInputData.map(x => {
+      val record = new GenericData.Record(schema.toAvroSchema) {{ put("date", x) }}
+      converter(record).get(0)
+    })
+
+    assert(dateOutputData(0).toString === LocalDate.ofEpochDay(dateInputData(0)).toString)
+    assert(dateOutputData(1).toString === LocalDate.ofEpochDay(dateInputData(1)).toString)
+    assert(dateOutputData(2).toString === LocalDate.ofEpochDay(dateInputData(2)).toString)
+  }
 }


(hudi) 10/16: fix(flink): add Apache license header to muttley/README.md (#18713)

Posted by vh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

voonhous pushed a commit to branch release-1.2.1
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 062e4c6ca5788aaba7ab821e5fc662f4f89f8c1d
Author: Xinli Shang <sh...@uber.com>
AuthorDate: Sun May 10 09:26:06 2026 -0700

    fix(flink): add Apache license header to muttley/README.md (#18713)
    
    PR #18394 added hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/muttley/README.md
    without an Apache license header, causing apache-rat:check to fail on every
    new build of master ("Too many files with unapproved license: 1").
    
    Prepend the standard Apache 2.0 HTML-comment license header so RAT passes.
    
    Verified locally:
      cd hudi-flink-datasource/hudi-flink
      mvn -Pflink1.20 -Dscala-2.12 apache-rat:check
      -> Rat check: Summary over all files. Unapproved: 0
    
    Co-authored-by: Xinli Shang <sh...@apache.org>
    (cherry picked from commit f2f6203aa496cd5e502690cc7fea913acc93dd61)
---
 .../main/java/org/apache/hudi/sink/muttley/README.md    | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/muttley/README.md b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/muttley/README.md
index 04c2e31d0e1e..1f0387cfdb95 100644
--- a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/muttley/README.md
+++ b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/muttley/README.md
@@ -1,3 +1,20 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
 # Internal Uber Components (Optional)
 
 This package contains integration with internal Uber services and is **optional** for Apache Hudi users.