You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/09/03 18:34:13 UTC

[PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

yihua opened a new pull request, #19839:
URL: https://github.com/apache/hudi/pull/19839

   ### Describe the issue this Pull Request addresses
   
   issue: #19838
   
   Adds the regression test only. The behavior fix is deliberately not included, so the test is added disabled.
   
   ### Summary and Changelog
   
   A record whose ordering field value is null is accepted into the write instead of being routed to the error table as a `RECORD_CREATION` failure. The existing `TestHoodieStreamerUtils` never entered the ordering-value path at all, because it configures no ordering field, so the gap was uncovered.
   
   Adds `testCreateHoodieRecordsWithNullOrderingValue` to `TestHoodieStreamerUtils`, parameterized over both record-creation paths and selected through `hoodie.write.merge.handle.class`: the default `FileGroupReaderBasedMergeHandle` for the file-group-reader path, and `HoodieMergeHandle` for the payload path. It asserts the record is absent from the output and present in the error table as a `RECORD_CREATION` event, with the full expected payload rather than a count.
   
   The test is `@Disabled` because it asserts the intended behavior and both paths currently fail it. Enabling it belongs with the change that makes record creation reject a null ordering value.
   
   Verified on this base with the annotation removed: both parameters fail with
   
   ```
   expected: <[]> but was: <[HoodieRecord{key=HoodieKey(recordKey=key1, partitionPath=path1), currentLocation='null', newLocation='null'}]>
   ```
   
   and with a reject-on-null added at record creation, both pass. The expected error-table payload was measured rather than assumed: a schema-valid record is serialized by `HoodieAvroUtils.safeAvroToJsonString` through the Avro `JsonEncoder`, which wraps nullable union values. The pre-existing `testCreateHoodieRecordsWithError` asserts an unwrapped payload only because its record is schema-invalid, so serialization throws and falls back to `record.toString()`.
   
   ### Impact
   
   None. Test-only, and the added test is disabled.
   
   ### Risk Level
   
   none
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5530523863

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b44b55ab342f1a84afc6f4acedc106d30445fe81 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5531674228

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813",
       "triggerID" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b44b55ab342f1a84afc6f4acedc106d30445fe81 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5937132781

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813",
       "triggerID" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17410",
       "triggerID" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "42099f22ebf34f86c416f7eb0168c85a06ee4bf1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "42099f22ebf34f86c416f7eb0168c85a06ee4bf1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17410) 
   * 42099f22ebf34f86c416f7eb0168c85a06ee4bf1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5927376123

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813",
       "triggerID" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17410",
       "triggerID" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b44b55ab342f1a84afc6f4acedc106d30445fe81 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813) 
   * fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17410) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5952438731

   Okay, just finished reading through this PR:
   1. It tests the streamer is able to quarantine bad source records to an error table instead of writing it to bad tables.
   2. It covers bad records across a few write modes
       - insert
       - bulk_insert
       - upsert
   3. The implementation for what the error table writer is controlled by`hoodie.errortable.write.class`
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #19839: test(utilities): cover quarantining a record with a null ordering value
URL: https://github.com/apache/hudi/pull/19839


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19839:
URL: https://github.com/apache/hudi/pull/19839#discussion_r4161260763


##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamerErrorTableWriteFlow.java:
##########
@@ -19,75 +19,223 @@
 
 package org.apache.hudi.utilities.deltastreamer;
 
+import org.apache.hudi.client.WriteStatus;
 import org.apache.hudi.common.model.WriteOperationType;
 import org.apache.hudi.common.table.HoodieTableMetaClient;
 import org.apache.hudi.common.util.collection.Tuple3;
 import org.apache.hudi.hadoop.fs.HadoopFSUtils;
+import org.apache.hudi.utilities.streamer.ErrorEvent;
 
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
 import org.apache.hadoop.fs.Path;
 import org.apache.spark.sql.Dataset;
 import org.apache.spark.sql.Row;
 import org.apache.spark.sql.functions;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.types.StructType;
 import org.junit.jupiter.params.ParameterizedTest;
 import org.junit.jupiter.params.provider.Arguments;
 import org.junit.jupiter.params.provider.MethodSource;
 
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Predicate;
+import java.util.function.UnaryOperator;
 import java.util.stream.Stream;
 
 import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
 
 class TestHoodieDeltaStreamerErrorTableWriteFlow extends TestHoodieDeltaStreamerSchemaEvolutionBase {
+  private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
+
+  /**
+   * Bad records a source batch can carry, each built from valid generated records by one column mutation. A case
+   * declares, per write operation, whether the streamer must route the record to the error table or write it to the
+   * base table, and the reason it is routed there, so every case is exercised by the same streamer sync. Cover a new
+   * failure mode by adding a case here rather than a new test, since each streamer sync is expensive.
+   */
+  enum BadRecordCase {
+    // The original record key is kept in driver, so each quarantined record stays attributable.
+    EMPTY_RECORD_KEY(df -> df.withColumn("driver", functions.col("_row_key")).withColumn("_row_key", functions.lit("")),
+        operation -> true, ErrorEvent.ErrorReason.RECORD_CREATION),
+    // Only an upsert combines records by their ordering value; the other operations write a null one through.
+    NULL_ORDERING_VALUE(df -> df.withColumn("timestamp", functions.lit(null).cast(DataTypes.LongType)),
+        operation -> operation == WriteOperationType.UPSERT, ErrorEvent.ErrorReason.RECORD_CREATION);
+
+    private final UnaryOperator<Dataset<Row>> mutation;
+    private final Predicate<WriteOperationType> routedToErrorTable;
+    private final ErrorEvent.ErrorReason errorReason;
+
+    BadRecordCase(UnaryOperator<Dataset<Row>> mutation, Predicate<WriteOperationType> routedToErrorTable,
+                  ErrorEvent.ErrorReason errorReason) {
+      this.mutation = mutation;
+      this.routedToErrorTable = routedToErrorTable;
+      this.errorReason = errorReason;
+    }
+  }
+
   protected void testBase(Tuple3<Integer, Integer, Integer> sourceGenInfo) throws Exception {
     int totalRecords = sourceGenInfo.f0;
     int errorRecords = sourceGenInfo.f1;
     int numFiles = sourceGenInfo.f2;
     boolean shouldCreateMultipleSourceFiles = numFiles > 1;
+    // Expected outcome, kept up to date as each batch is built: the base table as record key to rider, and the records
+    // routed to the error table as (record, reason), where a record is named by its key, or by driver when the key is empty.
+    Map<String, String> expectedRiders = new HashMap<>();
+    List<String> expectedErrorRecords = new ArrayList<>();
 
     PARQUET_SOURCE_ROOT = basePath + "parquetFilesDfs" + testNum++;
 
+    List<Row> validRecords = Collections.emptyList();
+    StructType sourceSchema = null;
     if (totalRecords > 0) {
       if (shouldCreateMultipleSourceFiles) {
         prepareParquetDFSMultiFiles(totalRecords - errorRecords, PARQUET_SOURCE_ROOT, numFiles);
       } else {
         prepareParquetDFSFiles(totalRecords - errorRecords, PARQUET_SOURCE_ROOT);
       }
+      Dataset<Row> validRecordsDf = sparkSession.read().parquet(PARQUET_SOURCE_ROOT);
+      sourceSchema = validRecordsDf.schema();
+      validRecords = validRecordsDf.orderBy("_row_key").collectAsList();
+      validRecords.forEach(row -> expectedRiders.put(row.getAs("_row_key"), row.getAs("rider")));
 
-      // Add error data to the source
+      // Add errorRecords records of each bad-record case to the source
       if (errorRecords > 0) {
-        String errorDataSourceRoot = basePath + "parquetErrorFilesDfs" + testNum++;
-        prepareParquetDFSFiles(errorRecords, errorDataSourceRoot);
-        Dataset<Row> df = sparkSession.read().parquet(errorDataSourceRoot);
-        df = df.withColumn("_row_key", functions.lit(""));
-        // add error records to PARQUET_SOURCE_ROOT
-        addParquetData(df, false);
+        for (BadRecordCase badRecordCase : BadRecordCase.values()) {
+          String errorDataSourceRoot = basePath + "parquetErrorFilesDfs" + testNum++;
+          prepareParquetDFSFiles(errorRecords, errorDataSourceRoot);
+          addBadRecords(badRecordCase, sparkSession.read().parquet(errorDataSourceRoot), expectedRiders, expectedErrorRecords);
+        }
       }
     } else {
       fs.mkdirs(new Path(PARQUET_SOURCE_ROOT));
     }
 
     tableName = "test_parquet_table" + testNum;
     tableBasePath = basePath + tableName;
-    this.deltaStreamer = new HoodieDeltaStreamer(getDeltaStreamerConfig(), jsc);
+    HoodieDeltaStreamer.Config cfg = getDeltaStreamerConfig();
+    cfg.operation = writeOperationType;
+    this.deltaStreamer = new HoodieDeltaStreamer(cfg, jsc);
     this.deltaStreamer.sync();
+    int expectedInstants = 1;
+    int lastSyncErrorRecords = expectedErrorRecords.size();
+
+    // An upsert also updates stored records, in a second sync: a valid update must be applied, every bad record in the
+    // batch must reach the error table, and a null ordering value on a stored record must leave that record unchanged.
+    // Without a guard at record creation, that null ordering value reaches the record merger. Every check runs after
+    // the last sync, so a failure in the first one does not stop the second from reaching the merger.
+    if (writeOperationType == WriteOperationType.UPSERT && errorRecords > 0) {
+      assertTrue(validRecords.size() >= (BadRecordCase.values().length + 1) * errorRecords,
+          "the second sync updates errorRecords stored records per bad-record case, plus as many valid updates");
+      int firstSyncErrorRecords = expectedErrorRecords.size();
+      List<Row> validUpdates = updatesOf(validRecords.subList(0, errorRecords), sourceSchema);
+      addParquetData(validUpdates, sourceSchema);
+      validUpdates.forEach(row -> expectedRiders.put(row.getAs("_row_key"), row.getAs("rider")));
+      BadRecordCase[] badRecordCases = BadRecordCase.values();
+      for (int i = 0; i < badRecordCases.length; i++) {
+        List<Row> storedRecords = validRecords.subList((i + 1) * errorRecords, (i + 2) * errorRecords);
+        addBadRecords(badRecordCases[i], sparkSession.createDataFrame(updatesOf(storedRecords, sourceSchema), sourceSchema),
+            expectedRiders, expectedErrorRecords);
+      }
+      this.deltaStreamer = new HoodieDeltaStreamer(cfg, jsc);
+      this.deltaStreamer.sync();
+      expectedInstants++;
+      lastSyncErrorRecords = expectedErrorRecords.size() - firstSyncErrorRecords;
+    }
 
     // base table validation
     Dataset<Row> baseDf = sparkSession.read().format("hudi").load(tableBasePath);
-    assertEquals(totalRecords - errorRecords, baseDf.count());
+    if (expectedRiders.isEmpty()) {
+      assertEquals(0, baseDf.count());
+    } else {
+      Map<String, String> actualRiders = new HashMap<>();
+      baseDf.select("_row_key", "rider").collectAsList().forEach(row -> actualRiders.put(row.getString(0), row.getString(1)));
+      assertEquals(expectedRiders, actualRiders);
+    }
 
     HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
         .setConf(HadoopFSUtils.getStorageConfWithCopy(jsc.hadoopConfiguration()))
         .setBasePath(tableBasePath).build();
-    assertEquals(1, metaClient.getActiveTimeline().getInstants().size());
+    assertEquals(expectedInstants, metaClient.getActiveTimeline().getInstants().size());
 
     // error table validation
     if (withErrorTable) {
-      TestHoodieDeltaStreamerSchemaEvolutionExtensive.validateErrorTable(errorRecords, this.writeErrorTableInParallelWithBaseTable);
+      Collections.sort(expectedErrorRecords);
+      List<Object> receivedErrorEvents = new ArrayList<>();
+      TestErrorTable.receivedErrorEvents.forEach(errorEvents -> receivedErrorEvents.addAll(errorEvents.collect()));
+      assertEquals(expectedErrorRecords, errorRecordsOf(receivedErrorEvents));
+      // What the error table writer committed, not only what it was handed
+      List<Object> committed = new ArrayList<>();
+      TestErrorTable.commited.values().forEach(errors -> errors.ifPresent(rdd -> committed.addAll(rdd.collect())));
+      if (this.writeErrorTableInParallelWithBaseTable) {
+        // The unified write commits write statuses, and each commit replaces the previous one

Review Comment:
   πŸ€– nit: `addParquetData` is now overloaded with unrelated signatures (Dataset+boolean vs List+schema that just wraps into a DataFrame) β€” might be worth a distinct name like `addParquetRows` to avoid confusing call sites.
   
   <sub><i>⚠️ AI-generated; verify before applying. React πŸ‘/πŸ‘Ž to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5938413822

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813",
       "triggerID" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17410",
       "triggerID" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "42099f22ebf34f86c416f7eb0168c85a06ee4bf1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17413",
       "triggerID" : "42099f22ebf34f86c416f7eb0168c85a06ee4bf1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 42099f22ebf34f86c416f7eb0168c85a06ee4bf1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17413) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5937253559

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813",
       "triggerID" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17410",
       "triggerID" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "42099f22ebf34f86c416f7eb0168c85a06ee4bf1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17413",
       "triggerID" : "42099f22ebf34f86c416f7eb0168c85a06ee4bf1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17410) 
   * 42099f22ebf34f86c416f7eb0168c85a06ee4bf1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17413) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-6701383516

   Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5952476456

   @yihua Can you please fix the failing CIs, restarted a few times and this test prolly caught a regression. Either pin the regression behaviour and explicitly call it out as incorrect which will be fixed in a separate issue, or fix it together with this issue?
   
   Another thing is, this test only covers AVRO record type, can we also test the SPARK record type branch? Or is that intentionally not covered?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5928330502

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813",
       "triggerID" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17410",
       "triggerID" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17410) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5530602903

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813",
       "triggerID" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b44b55ab342f1a84afc6f4acedc106d30445fe81 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5530567294

   ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19839?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 78.17%. Comparing base ([`31904ac`](https://app.codecov.io/gh/apache/hudi/commit/31904acd2a1d3a215b6bd12e73fb81c9423a71e8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b44b55a`](https://app.codecov.io/gh/apache/hudi/commit/b44b55ab342f1a84afc6f4acedc106d30445fe81?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 4 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #19839      +/-   ##
   ============================================
   - Coverage     78.31%   78.17%   -0.15%     
   - Complexity    33894    34007     +113     
   ============================================
     Files          2542     2543       +1     
     Lines        141775   142533     +758     
     Branches      17197    17430     +233     
   ============================================
   + Hits         111031   111419     +388     
   - Misses        23032    23335     +303     
   - Partials       7712     7779      +67     
   ```
   
   | [Components](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ” | |
   |---|---|---|
   | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.93% <ΓΈ> (-0.71%)` | :arrow_down: |
   | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.21% <ΓΈ> (+<0.01%)` | :arrow_up: |
   | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.66% <ΓΈ> (ΓΈ)` | |
   | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.24% <ΓΈ> (ΓΈ)` | |
   | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.51% <ΓΈ> (ΓΈ)` | |
   | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ΓΈ> (ΓΈ)` | |
   | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.72% <ΓΈ> (ΓΈ)` | |
   | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.56% <ΓΈ> (ΓΈ)` | |
   | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.93% <ΓΈ> (ΓΈ)` | |
   | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ΓΈ> (ΓΈ)` | |
   | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.81% <ΓΈ> (ΓΈ)` | |
   | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19839/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.96% <ΓΈ> (ΓΈ)` | |
   
   | [Flag](https://app.codecov.io/gh/apache/hudi/pull/19839/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ” | |
   |---|---|---|
   | [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19839/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.11% <ΓΈ> (+0.04%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   [see 9 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19839/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] test(utilities): cover quarantining a record with a null ordering value [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19839:
URL: https://github.com/apache/hudi/pull/19839#issuecomment-5927282093

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813",
       "triggerID" : "b44b55ab342f1a84afc6f4acedc106d30445fe81",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b44b55ab342f1a84afc6f4acedc106d30445fe81 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16813) 
   * fcc731edf9d0cc3d62d5f4cd2824ec20d36736f4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org