You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/04/16 20:41:26 UTC

[PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

the-other-tim-brown opened a new pull request, #13157:
URL: https://github.com/apache/hudi/pull/13157

   ### Change Logs
   
   - Fix issue in HoodieTestDataGenerator where multiple updates were created for the same key
   - Update `TestHoodieFileGroupReaderBase` to handle the comparison logic and to use the generated records as the expected values instead of re-reading the values. In the future, there will not be a non-FGReader path so we need to anticipate that.
   - Add placeholders for setting up tests where `populateMetaFields` is set to false
   
   ### Impact
   
   Splits out the refactoring of our tests into its own PR to cut down on the size of the code changes. This also adds data validations to tests whereas previous counts were being used and only a single file slice was being queried.
   
   ### Risk level (write none, low medium or high below)
   
   None, this is strictly test improvements
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2810724350

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d598040cc912707f2bbd6df1fbe3d99d254386f2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #13157:
URL: https://github.com/apache/hudi/pull/13157


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2810790530

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d598040cc912707f2bbd6df1fbe3d99d254386f2 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904) 
   * e274262a5fbca0f3d4f163938b1748e275b2881b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2810787789

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d598040cc912707f2bbd6df1fbe3d99d254386f2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904) 
   * e274262a5fbca0f3d4f163938b1748e275b2881b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2813967853

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4906",
       "triggerID" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6148db477fb6bfb49a95fb58b9035ded100ff97c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6148db477fb6bfb49a95fb58b9035ded100ff97c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f52edb7163a6fc7067d78ecce973803d4c0e1d85 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4906) 
   * 6148db477fb6bfb49a95fb58b9035ded100ff97c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2810884807

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d598040cc912707f2bbd6df1fbe3d99d254386f2 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904) 
   * e274262a5fbca0f3d4f163938b1748e275b2881b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2811281403

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4906",
       "triggerID" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f52edb7163a6fc7067d78ecce973803d4c0e1d85 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4906) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13157:
URL: https://github.com/apache/hudi/pull/13157#discussion_r2049612332


##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -366,4 +400,70 @@ private boolean shouldValidatePartialRead(FileSlice fileSlice, Schema requestedS
     }
     return false;
   }
+
+  private List<HoodieRecord> mergeRecordLists(List<HoodieRecord> updates, List<HoodieRecord> existing) {
+    Set<String> updatedKeys = updates.stream().map(HoodieRecord::getRecordKey).collect(Collectors.toSet());
+    return Stream.concat(updates.stream(), existing.stream().filter(record -> !updatedKeys.contains(record.getRecordKey())))
+            .collect(Collectors.toList());
+  }
+
+  private List<RecordIdentifier> convertHoodieRecords(List<HoodieRecord> records, Schema schema) {
+    int keyFieldIndex = schema.getField(KEY_FIELD_NAME).pos();
+    int orderingFieldIndex = schema.getField(PRECOMBINE_FIELD_NAME).pos();
+    int partitionFieldIndex = schema.getField(PARTITION_FIELD_NAME).pos();
+    int riderFieldIndex = schema.getField(RIDER_FIELD_NAME).pos();
+    return records.stream().map(record -> {
+      try {
+        RawTripTestPayload payload = (RawTripTestPayload) record.getData();
+        IndexedRecord indexedRecord = payload.getRecordToInsert(schema);
+        return new RecordIdentifier(indexedRecord.get(keyFieldIndex).toString(), indexedRecord.get(orderingFieldIndex).toString(),
+            indexedRecord.get(partitionFieldIndex).toString(), indexedRecord.get(riderFieldIndex).toString());
+      } catch (IOException e) {
+        throw new UncheckedIOException(e);
+      }
+    }).collect(Collectors.toList());
+  }
+
+  private List<RecordIdentifier> convertEngineRecords(List<T> records, Schema schema, HoodieReaderContext<T> readerContext) {
+    return records.stream()
+        .map(record -> new RecordIdentifier(readerContext.getValue(record, schema, KEY_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, PRECOMBINE_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, PARTITION_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, RIDER_FIELD_NAME).toString()))
+        .collect(Collectors.toList());
+  }
+
+  /**
+   * Used for equality checks between the expected and actual records for the FileGroupReader.
+   */
+  private static class RecordIdentifier {

Review Comment:
   Updated the code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13157:
URL: https://github.com/apache/hudi/pull/13157#discussion_r2049452602


##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -366,4 +400,70 @@ private boolean shouldValidatePartialRead(FileSlice fileSlice, Schema requestedS
     }
     return false;
   }
+
+  private List<HoodieRecord> mergeRecordLists(List<HoodieRecord> updates, List<HoodieRecord> existing) {
+    Set<String> updatedKeys = updates.stream().map(HoodieRecord::getRecordKey).collect(Collectors.toSet());
+    return Stream.concat(updates.stream(), existing.stream().filter(record -> !updatedKeys.contains(record.getRecordKey())))
+            .collect(Collectors.toList());
+  }
+
+  private List<RecordIdentifier> convertHoodieRecords(List<HoodieRecord> records, Schema schema) {
+    int keyFieldIndex = schema.getField(KEY_FIELD_NAME).pos();
+    int orderingFieldIndex = schema.getField(PRECOMBINE_FIELD_NAME).pos();
+    int partitionFieldIndex = schema.getField(PARTITION_FIELD_NAME).pos();
+    int riderFieldIndex = schema.getField(RIDER_FIELD_NAME).pos();
+    return records.stream().map(record -> {
+      try {
+        RawTripTestPayload payload = (RawTripTestPayload) record.getData();
+        IndexedRecord indexedRecord = payload.getRecordToInsert(schema);
+        return new RecordIdentifier(indexedRecord.get(keyFieldIndex).toString(), indexedRecord.get(orderingFieldIndex).toString(),
+            indexedRecord.get(partitionFieldIndex).toString(), indexedRecord.get(riderFieldIndex).toString());
+      } catch (IOException e) {
+        throw new UncheckedIOException(e);
+      }
+    }).collect(Collectors.toList());
+  }
+
+  private List<RecordIdentifier> convertEngineRecords(List<T> records, Schema schema, HoodieReaderContext<T> readerContext) {
+    return records.stream()
+        .map(record -> new RecordIdentifier(readerContext.getValue(record, schema, KEY_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, PRECOMBINE_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, PARTITION_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, RIDER_FIELD_NAME).toString()))
+        .collect(Collectors.toList());
+  }
+
+  /**
+   * Used for equality checks between the expected and actual records for the FileGroupReader.
+   */
+  private static class RecordIdentifier {

Review Comment:
   do you think we can move this to HoodieTestDataGenerator only and also, converting from HoodieRecord to RecordIndentifier so that we can start using it across the board for test assertions. 



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java:
##########
@@ -952,10 +952,15 @@ public List<HoodieRecord> generateUpdates(String instantTime, Integer n) throws
    */
   public List<HoodieRecord> generateUpdates(String instantTime, Integer n, String schemaStr) throws IOException {
     List<HoodieRecord> updates = new ArrayList<>();
+    Map<Integer, KeyPartition> existingKeys = existingKeysBySchema.get(schemaStr);

Review Comment:
   we do have another method called generateUniqueUpdates. may be we can dedup and retain just 1 method. 



##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -366,4 +400,70 @@ private boolean shouldValidatePartialRead(FileSlice fileSlice, Schema requestedS
     }
     return false;
   }
+
+  private List<HoodieRecord> mergeRecordLists(List<HoodieRecord> updates, List<HoodieRecord> existing) {
+    Set<String> updatedKeys = updates.stream().map(HoodieRecord::getRecordKey).collect(Collectors.toSet());
+    return Stream.concat(updates.stream(), existing.stream().filter(record -> !updatedKeys.contains(record.getRecordKey())))
+            .collect(Collectors.toList());
+  }
+
+  private List<RecordIdentifier> convertHoodieRecords(List<HoodieRecord> records, Schema schema) {
+    int keyFieldIndex = schema.getField(KEY_FIELD_NAME).pos();
+    int orderingFieldIndex = schema.getField(PRECOMBINE_FIELD_NAME).pos();
+    int partitionFieldIndex = schema.getField(PARTITION_FIELD_NAME).pos();
+    int riderFieldIndex = schema.getField(RIDER_FIELD_NAME).pos();
+    return records.stream().map(record -> {
+      try {
+        RawTripTestPayload payload = (RawTripTestPayload) record.getData();
+        IndexedRecord indexedRecord = payload.getRecordToInsert(schema);
+        return new RecordIdentifier(indexedRecord.get(keyFieldIndex).toString(), indexedRecord.get(orderingFieldIndex).toString(),
+            indexedRecord.get(partitionFieldIndex).toString(), indexedRecord.get(riderFieldIndex).toString());
+      } catch (IOException e) {
+        throw new UncheckedIOException(e);
+      }
+    }).collect(Collectors.toList());
+  }
+
+  private List<RecordIdentifier> convertEngineRecords(List<T> records, Schema schema, HoodieReaderContext<T> readerContext) {
+    return records.stream()
+        .map(record -> new RecordIdentifier(readerContext.getValue(record, schema, KEY_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, PRECOMBINE_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, PARTITION_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, RIDER_FIELD_NAME).toString()))
+        .collect(Collectors.toList());
+  }
+
+  /**
+   * Used for equality checks between the expected and actual records for the FileGroupReader.
+   */
+  private static class RecordIdentifier {
+    private final String recordKey;
+    private final String orderingVal;
+    private final String partitionPath;
+    private final String riderValue;

Review Comment:
   if I am not wrong, for a given batch of records generated using HoodieTestDataGenerator, we will have same value for rider. Can we also pick "fare" column may be since its expected to have diff values for diff records. 
   not too strong on this suggestion. will let you take a call. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2811018211

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4906",
       "triggerID" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e274262a5fbca0f3d4f163938b1748e275b2881b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905) 
   * f52edb7163a6fc7067d78ecce973803d4c0e1d85 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4906) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2814085460

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4906",
       "triggerID" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6148db477fb6bfb49a95fb58b9035ded100ff97c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4931",
       "triggerID" : "6148db477fb6bfb49a95fb58b9035ded100ff97c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6148db477fb6bfb49a95fb58b9035ded100ff97c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4931) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2813970239

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4906",
       "triggerID" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6148db477fb6bfb49a95fb58b9035ded100ff97c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4931",
       "triggerID" : "6148db477fb6bfb49a95fb58b9035ded100ff97c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f52edb7163a6fc7067d78ecce973803d4c0e1d85 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4906) 
   * 6148db477fb6bfb49a95fb58b9035ded100ff97c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4931) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2810729118

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d598040cc912707f2bbd6df1fbe3d99d254386f2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2810951262

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e274262a5fbca0f3d4f163938b1748e275b2881b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905) 
   * f52edb7163a6fc7067d78ecce973803d4c0e1d85 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13157:
URL: https://github.com/apache/hudi/pull/13157#discussion_r2049519640


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java:
##########
@@ -952,10 +952,15 @@ public List<HoodieRecord> generateUpdates(String instantTime, Integer n) throws
    */
   public List<HoodieRecord> generateUpdates(String instantTime, Integer n, String schemaStr) throws IOException {
     List<HoodieRecord> updates = new ArrayList<>();
+    Map<Integer, KeyPartition> existingKeys = existingKeysBySchema.get(schemaStr);

Review Comment:
   There are some cases where there are multiple updates for the same key expected, like when testing deduping within the ingestion. I will revert the changes here and just use the other method for these FGReader tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13157:
URL: https://github.com/apache/hudi/pull/13157#discussion_r2049517031


##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -366,4 +400,70 @@ private boolean shouldValidatePartialRead(FileSlice fileSlice, Schema requestedS
     }
     return false;
   }
+
+  private List<HoodieRecord> mergeRecordLists(List<HoodieRecord> updates, List<HoodieRecord> existing) {
+    Set<String> updatedKeys = updates.stream().map(HoodieRecord::getRecordKey).collect(Collectors.toSet());
+    return Stream.concat(updates.stream(), existing.stream().filter(record -> !updatedKeys.contains(record.getRecordKey())))
+            .collect(Collectors.toList());
+  }
+
+  private List<RecordIdentifier> convertHoodieRecords(List<HoodieRecord> records, Schema schema) {
+    int keyFieldIndex = schema.getField(KEY_FIELD_NAME).pos();
+    int orderingFieldIndex = schema.getField(PRECOMBINE_FIELD_NAME).pos();
+    int partitionFieldIndex = schema.getField(PARTITION_FIELD_NAME).pos();
+    int riderFieldIndex = schema.getField(RIDER_FIELD_NAME).pos();
+    return records.stream().map(record -> {
+      try {
+        RawTripTestPayload payload = (RawTripTestPayload) record.getData();
+        IndexedRecord indexedRecord = payload.getRecordToInsert(schema);
+        return new RecordIdentifier(indexedRecord.get(keyFieldIndex).toString(), indexedRecord.get(orderingFieldIndex).toString(),
+            indexedRecord.get(partitionFieldIndex).toString(), indexedRecord.get(riderFieldIndex).toString());
+      } catch (IOException e) {
+        throw new UncheckedIOException(e);
+      }
+    }).collect(Collectors.toList());
+  }
+
+  private List<RecordIdentifier> convertEngineRecords(List<T> records, Schema schema, HoodieReaderContext<T> readerContext) {
+    return records.stream()
+        .map(record -> new RecordIdentifier(readerContext.getValue(record, schema, KEY_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, PRECOMBINE_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, PARTITION_FIELD_NAME).toString(),
+            readerContext.getValue(record, schema, RIDER_FIELD_NAME).toString()))
+        .collect(Collectors.toList());
+  }
+
+  /**
+   * Used for equality checks between the expected and actual records for the FileGroupReader.
+   */
+  private static class RecordIdentifier {
+    private final String recordKey;
+    private final String orderingVal;
+    private final String partitionPath;
+    private final String riderValue;

Review Comment:
   the record key will be different for each row so that covers the uniqueness issue. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-9316] HoodieFileGroupReader: Refactor tests to support virtual keys [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13157:
URL: https://github.com/apache/hudi/pull/13157#issuecomment-2810949004

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904",
       "triggerID" : "d598040cc912707f2bbd6df1fbe3d99d254386f2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905",
       "triggerID" : "e274262a5fbca0f3d4f163938b1748e275b2881b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f52edb7163a6fc7067d78ecce973803d4c0e1d85",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d598040cc912707f2bbd6df1fbe3d99d254386f2 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4904) 
   * e274262a5fbca0f3d4f163938b1748e275b2881b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4905) 
   * f52edb7163a6fc7067d78ecce973803d4c0e1d85 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org