You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/19 16:29:19 UTC

[GitHub] [hudi] manojpec opened a new pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

manojpec opened a new pull request #3825:
URL: https://github.com/apache/hudi/pull/3825


   ## What is the purpose of the pull request
   
   TestCleaner - keep latest version policy test with metadata table enabled
   
   ## Brief change log
   
    - HoodieTestTable::getTestTableStateWithPartitionFileInfo() is refactored to
      accept a map of partition to its files name and length list instead of
      just file count
   
    - FileCreateUtils::createCommit() is changed to accept an optional param
      HoodieCommitMetadata and existing tests using this tests are modified
      accordingly.
   
    - TestCleaner::testKeepLatestFileVersions is updated to use the HoodieMetadataTestTable
      and doWriteOperations() instead of addCommit().
   
   ## Verify this pull request
   
   *(Please pick either of the following options)*
   
   This pull request is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   *(example:)*
   
     - *Added integration tests for end-to-end.*
     - *Added HoodieClientWriteTest to verify the change.*
     - *Manually verified the change by running a job locally.*
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] manojpec commented on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
manojpec commented on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-948046614






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] manojpec commented on a change in pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
manojpec commented on a change in pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#discussion_r732976497



##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/TestCleaner.java
##########
@@ -635,47 +638,61 @@ private void testFailedInsertAndCleanByCommits(
   }
 
   /**
-   * Test HoodieTable.clean() Cleaning by versions logic.
+   * Test Hudi COW Table Cleaner - Keep the latest file versions policy.
    */
   @ParameterizedTest
   @ValueSource(booleans = {false, true})
   public void testKeepLatestFileVersions(Boolean enableBootstrapSourceClean) throws Exception {
     HoodieWriteConfig config =
         HoodieWriteConfig.newBuilder().withPath(basePath)
-            .withMetadataConfig(HoodieMetadataConfig.newBuilder().withAssumeDatePartitioning(true).enable(false).build())
+            .withMetadataConfig(HoodieMetadataConfig.newBuilder().withAssumeDatePartitioning(true).enable(true).build())
             .withCompactionConfig(HoodieCompactionConfig.newBuilder()
                 .withCleanBootstrapBaseFileEnabled(enableBootstrapSourceClean)
                 .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS).retainFileVersions(1).build())
             .build();
-    HoodieTestTable testTable = HoodieTestTable.of(metaClient);
-    String p0 = "2020/01/01";
-    String p1 = "2020/01/02";
-    Map<String, List<BootstrapFileMapping>> bootstrapMapping = enableBootstrapSourceClean ? generateBootstrapIndexAndSourceData(p0, p1) : null;
+
+    HoodieTableMetadataWriter metadataWriter = SparkHoodieBackedTableMetadataWriter.create(hadoopConf, config, context);
+    HoodieTestTable testTable = HoodieMetadataTestTable.of(metaClient, metadataWriter);
+
+    final String p0 = "2020/01/01";
+    final String p1 = "2020/01/02";
+    final Map<String, List<BootstrapFileMapping>> bootstrapMapping = enableBootstrapSourceClean
+        ? generateBootstrapIndexAndSourceData(p0, p1) : null;
 
     // make 1 commit, with 1 file per partition
-    String file1P0C0 = enableBootstrapSourceClean ? bootstrapMapping.get(p0).get(0).getFileId()
+    final String file1P0C0 = enableBootstrapSourceClean ? bootstrapMapping.get(p0).get(0).getFileId()
         : UUID.randomUUID().toString();
-    String file1P1C0 = enableBootstrapSourceClean ? bootstrapMapping.get(p1).get(0).getFileId()
+    final String file1P1C0 = enableBootstrapSourceClean ? bootstrapMapping.get(p1).get(0).getFileId()
         : UUID.randomUUID().toString();
-    testTable.addCommit("00000000000001").withBaseFilesInPartition(p0, file1P0C0).withBaseFilesInPartition(p1, file1P1C0);
+
+    Map<String, List<Pair<String, Integer>>> c1PartitionToFilesNameLengthMap = new HashMap<>();
+    c1PartitionToFilesNameLengthMap.put(p0, Collections.singletonList(Pair.of(file1P0C0, 100)));
+    c1PartitionToFilesNameLengthMap.put(p1, Collections.singletonList(Pair.of(file1P1C0, 200)));
+    testTable.doWriteOperation("00000000000001", WriteOperationType.INSERT, Arrays.asList(p0, p1),
+        c1PartitionToFilesNameLengthMap, false, false);
 
     List<HoodieCleanStat> hoodieCleanStatsOne = runCleaner(config);
     assertEquals(0, hoodieCleanStatsOne.size(), "Must not clean any files");
     assertTrue(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
     assertTrue(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
 
     // make next commit, with 1 insert & 1 update per partition
-    Map<String, String> partitionAndFileId002 = testTable.addCommit("00000000000002")
-        .withBaseFilesInPartition(p0, file1P0C0)
-        .withBaseFilesInPartition(p1, file1P1C0)
-        .getFileIdsWithBaseFilesInPartitions(p0, p1);
+    final String file2P0C1 = UUID.randomUUID().toString();
+    final String file2P1C1 = UUID.randomUUID().toString();
+    Map<String, List<Pair<String, Integer>>> c2PartitionToFilesNameLengthMap = new HashMap<>();
+    c2PartitionToFilesNameLengthMap.put(p0, Arrays.asList(Pair.of(file1P0C0, 101), Pair.of(file2P0C1, 100)));
+    c2PartitionToFilesNameLengthMap.put(p1, Arrays.asList(Pair.of(file1P1C0, 201), Pair.of(file2P1C1, 200)));
+    testTable.doWriteOperation("00000000000002", WriteOperationType.UPSERT, Collections.emptyList(),
+        c2PartitionToFilesNameLengthMap, false, false);
 
     List<HoodieCleanStat> hoodieCleanStatsTwo = runCleaner(config, 1);
-    // enableBootstrapSourceClean would delete the bootstrap base file as the same time
     HoodieCleanStat cleanStat = getCleanStat(hoodieCleanStatsTwo, p0);
+
+    // enableBootstrapSourceClean would delete the bootstrap base file at the same time

Review comment:
       fixed this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1325e87059640be4cb61f8112e7a4ccb80cd3142 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749",
       "triggerID" : "947804908",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751",
       "triggerID" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2752",
       "triggerID" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e5ba5cc8d251321d1420c2a87b9af478a517d3f4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2752) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d833f9b084264676d8ae95603d1e3c700cb493b8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749",
       "triggerID" : "947804908",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751",
       "triggerID" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 49f6c6bfd35edc58c95819a64815acc3beb616a5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751) 
   * e5ba5cc8d251321d1420c2a87b9af478a517d3f4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749",
       "triggerID" : "947804908",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751",
       "triggerID" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 49f6c6bfd35edc58c95819a64815acc3beb616a5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749",
       "triggerID" : "947804908",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751",
       "triggerID" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2752",
       "triggerID" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2755",
       "triggerID" : "948046677",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * e5ba5cc8d251321d1420c2a87b9af478a517d3f4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2752) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2755) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on a change in pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#discussion_r732809214



##########
File path: hudi-client/hudi-client-common/src/test/java/org/apache/hudi/common/testutils/HoodieMetadataTestTable.java
##########
@@ -56,11 +60,52 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient, HoodieTableMe
     return new HoodieMetadataTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient, writer);
   }
 
+  /**
+   * Add commits to the requested partitions and update metadata table.
+   *
+   * @param commitTime           - Commit time for the operation
+   * @param operationType        - Operation type
+   * @param newPartitionsToAdd   - New partitions to add for the operation
+   * @param partitions           - List of partitions for this operation
+   * @param filesPerPartition    - Total file count to create per partition
+   * @param bootstrap            - Whether bootstrapping needed for the operation
+   * @param createInflightCommit - Whether in flight commit needed for the operation
+   * @return Commit metadata for the commit operation performed.
+   * @throws Exception
+   */
   @Override
   public HoodieCommitMetadata doWriteOperation(String commitTime, WriteOperationType operationType,
                                                List<String> newPartitionsToAdd, List<String> partitions,
-                                               int filesPerPartition, boolean bootstrap, boolean createInflightCommit) throws Exception {
-    HoodieCommitMetadata commitMetadata = super.doWriteOperation(commitTime, operationType, newPartitionsToAdd, partitions, filesPerPartition, bootstrap, createInflightCommit);
+                                               int filesPerPartition, boolean bootstrap,
+                                               boolean createInflightCommit) throws Exception {
+    if (partitions.isEmpty()) {
+      partitions = Collections.singletonList(EMPTY_STRING);
+    }
+    Map<String, List<Pair<String, Integer>>> partitionToFilesNameLengthMap = getPartitionFiles(partitions,
+        filesPerPartition);
+    return this.doWriteOperation(commitTime, operationType, newPartitionsToAdd, partitionToFilesNameLengthMap,

Review comment:
       shouldn't we move this to HoodieTestTable. 
   We would like to keep only metadata writer related code here. 
   something like
   ```
   super.method()
   call writer if need be
   ```
   if writer is not involved, probably testTable is the right place to hold such methods.
   

##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/TestCleaner.java
##########
@@ -635,47 +638,61 @@ private void testFailedInsertAndCleanByCommits(
   }
 
   /**
-   * Test HoodieTable.clean() Cleaning by versions logic.
+   * Test Hudi COW Table Cleaner - Keep the latest file versions policy.
    */
   @ParameterizedTest
   @ValueSource(booleans = {false, true})
   public void testKeepLatestFileVersions(Boolean enableBootstrapSourceClean) throws Exception {
     HoodieWriteConfig config =
         HoodieWriteConfig.newBuilder().withPath(basePath)
-            .withMetadataConfig(HoodieMetadataConfig.newBuilder().withAssumeDatePartitioning(true).enable(false).build())
+            .withMetadataConfig(HoodieMetadataConfig.newBuilder().withAssumeDatePartitioning(true).enable(true).build())
             .withCompactionConfig(HoodieCompactionConfig.newBuilder()
                 .withCleanBootstrapBaseFileEnabled(enableBootstrapSourceClean)
                 .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS).retainFileVersions(1).build())
             .build();
-    HoodieTestTable testTable = HoodieTestTable.of(metaClient);
-    String p0 = "2020/01/01";
-    String p1 = "2020/01/02";
-    Map<String, List<BootstrapFileMapping>> bootstrapMapping = enableBootstrapSourceClean ? generateBootstrapIndexAndSourceData(p0, p1) : null;
+
+    HoodieTableMetadataWriter metadataWriter = SparkHoodieBackedTableMetadataWriter.create(hadoopConf, config, context);
+    HoodieTestTable testTable = HoodieMetadataTestTable.of(metaClient, metadataWriter);
+
+    final String p0 = "2020/01/01";
+    final String p1 = "2020/01/02";
+    final Map<String, List<BootstrapFileMapping>> bootstrapMapping = enableBootstrapSourceClean
+        ? generateBootstrapIndexAndSourceData(p0, p1) : null;
 
     // make 1 commit, with 1 file per partition
-    String file1P0C0 = enableBootstrapSourceClean ? bootstrapMapping.get(p0).get(0).getFileId()
+    final String file1P0C0 = enableBootstrapSourceClean ? bootstrapMapping.get(p0).get(0).getFileId()
         : UUID.randomUUID().toString();
-    String file1P1C0 = enableBootstrapSourceClean ? bootstrapMapping.get(p1).get(0).getFileId()
+    final String file1P1C0 = enableBootstrapSourceClean ? bootstrapMapping.get(p1).get(0).getFileId()
         : UUID.randomUUID().toString();
-    testTable.addCommit("00000000000001").withBaseFilesInPartition(p0, file1P0C0).withBaseFilesInPartition(p1, file1P1C0);
+
+    Map<String, List<Pair<String, Integer>>> c1PartitionToFilesNameLengthMap = new HashMap<>();
+    c1PartitionToFilesNameLengthMap.put(p0, Collections.singletonList(Pair.of(file1P0C0, 100)));
+    c1PartitionToFilesNameLengthMap.put(p1, Collections.singletonList(Pair.of(file1P1C0, 200)));
+    testTable.doWriteOperation("00000000000001", WriteOperationType.INSERT, Arrays.asList(p0, p1),
+        c1PartitionToFilesNameLengthMap, false, false);
 
     List<HoodieCleanStat> hoodieCleanStatsOne = runCleaner(config);
     assertEquals(0, hoodieCleanStatsOne.size(), "Must not clean any files");
     assertTrue(testTable.baseFileExists(p0, "00000000000001", file1P0C0));
     assertTrue(testTable.baseFileExists(p1, "00000000000001", file1P1C0));
 
     // make next commit, with 1 insert & 1 update per partition
-    Map<String, String> partitionAndFileId002 = testTable.addCommit("00000000000002")
-        .withBaseFilesInPartition(p0, file1P0C0)
-        .withBaseFilesInPartition(p1, file1P1C0)
-        .getFileIdsWithBaseFilesInPartitions(p0, p1);
+    final String file2P0C1 = UUID.randomUUID().toString();
+    final String file2P1C1 = UUID.randomUUID().toString();
+    Map<String, List<Pair<String, Integer>>> c2PartitionToFilesNameLengthMap = new HashMap<>();
+    c2PartitionToFilesNameLengthMap.put(p0, Arrays.asList(Pair.of(file1P0C0, 101), Pair.of(file2P0C1, 100)));
+    c2PartitionToFilesNameLengthMap.put(p1, Arrays.asList(Pair.of(file1P1C0, 201), Pair.of(file2P1C1, 200)));
+    testTable.doWriteOperation("00000000000002", WriteOperationType.UPSERT, Collections.emptyList(),
+        c2PartitionToFilesNameLengthMap, false, false);
 
     List<HoodieCleanStat> hoodieCleanStatsTwo = runCleaner(config, 1);
-    // enableBootstrapSourceClean would delete the bootstrap base file as the same time
     HoodieCleanStat cleanStat = getCleanStat(hoodieCleanStatsTwo, p0);
+
+    // enableBootstrapSourceClean would delete the bootstrap base file at the same time

Review comment:
       guess this comment should go just before line 689




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d833f9b084264676d8ae95603d1e3c700cb493b8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723) 
   * 1325e87059640be4cb61f8112e7a4ccb80cd3142 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d833f9b084264676d8ae95603d1e3c700cb493b8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d833f9b084264676d8ae95603d1e3c700cb493b8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723) 
   * 1325e87059640be4cb61f8112e7a4ccb80cd3142 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] manojpec commented on a change in pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
manojpec commented on a change in pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#discussion_r732970044



##########
File path: hudi-client/hudi-client-common/src/test/java/org/apache/hudi/common/testutils/HoodieMetadataTestTable.java
##########
@@ -56,11 +60,52 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient, HoodieTableMe
     return new HoodieMetadataTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient, writer);
   }
 
+  /**
+   * Add commits to the requested partitions and update metadata table.
+   *
+   * @param commitTime           - Commit time for the operation
+   * @param operationType        - Operation type
+   * @param newPartitionsToAdd   - New partitions to add for the operation
+   * @param partitions           - List of partitions for this operation
+   * @param filesPerPartition    - Total file count to create per partition
+   * @param bootstrap            - Whether bootstrapping needed for the operation
+   * @param createInflightCommit - Whether in flight commit needed for the operation
+   * @return Commit metadata for the commit operation performed.
+   * @throws Exception
+   */
   @Override
   public HoodieCommitMetadata doWriteOperation(String commitTime, WriteOperationType operationType,
                                                List<String> newPartitionsToAdd, List<String> partitions,
-                                               int filesPerPartition, boolean bootstrap, boolean createInflightCommit) throws Exception {
-    HoodieCommitMetadata commitMetadata = super.doWriteOperation(commitTime, operationType, newPartitionsToAdd, partitions, filesPerPartition, bootstrap, createInflightCommit);
+                                               int filesPerPartition, boolean bootstrap,
+                                               boolean createInflightCommit) throws Exception {
+    if (partitions.isEmpty()) {
+      partitions = Collections.singletonList(EMPTY_STRING);
+    }
+    Map<String, List<Pair<String, Integer>>> partitionToFilesNameLengthMap = getPartitionFiles(partitions,
+        filesPerPartition);
+    return this.doWriteOperation(commitTime, operationType, newPartitionsToAdd, partitionToFilesNameLengthMap,

Review comment:
       The derived HoodieMetadataTestTable has overridden two of doWriteOperation() methods. There are tests which are using both of these versions. `super.doWriteOperation()` does proxies the call to few more hops and ultimately calls back the other doWriteOperation(), which is again overridden in HoodieMetadataTestTable. This ended up making two HoodieTableMetadataWriter::update() call. We need to avoid back and forth hopping between  HoodieMetadataTestTable and HoodieTestTable for the same method. So, did the current model. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749",
       "triggerID" : "947804908",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751",
       "triggerID" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2752",
       "triggerID" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2755",
       "triggerID" : "948046677",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * e5ba5cc8d251321d1420c2a87b9af478a517d3f4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2752) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2755) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] manojpec commented on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
manojpec commented on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-947804908


   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749",
       "triggerID" : "947804908",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 1325e87059640be4cb61f8112e7a4ccb80cd3142 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] manojpec commented on a change in pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
manojpec commented on a change in pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#discussion_r733019287



##########
File path: hudi-client/hudi-client-common/src/test/java/org/apache/hudi/common/testutils/HoodieMetadataTestTable.java
##########
@@ -56,11 +60,52 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient, HoodieTableMe
     return new HoodieMetadataTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient, writer);
   }
 
+  /**
+   * Add commits to the requested partitions and update metadata table.
+   *
+   * @param commitTime           - Commit time for the operation
+   * @param operationType        - Operation type
+   * @param newPartitionsToAdd   - New partitions to add for the operation
+   * @param partitions           - List of partitions for this operation
+   * @param filesPerPartition    - Total file count to create per partition
+   * @param bootstrap            - Whether bootstrapping needed for the operation
+   * @param createInflightCommit - Whether in flight commit needed for the operation
+   * @return Commit metadata for the commit operation performed.
+   * @throws Exception
+   */
   @Override
   public HoodieCommitMetadata doWriteOperation(String commitTime, WriteOperationType operationType,
                                                List<String> newPartitionsToAdd, List<String> partitions,
-                                               int filesPerPartition, boolean bootstrap, boolean createInflightCommit) throws Exception {
-    HoodieCommitMetadata commitMetadata = super.doWriteOperation(commitTime, operationType, newPartitionsToAdd, partitions, filesPerPartition, bootstrap, createInflightCommit);
+                                               int filesPerPartition, boolean bootstrap,
+                                               boolean createInflightCommit) throws Exception {
+    if (partitions.isEmpty()) {
+      partitions = Collections.singletonList(EMPTY_STRING);
+    }
+    Map<String, List<Pair<String, Integer>>> partitionToFilesNameLengthMap = getPartitionFiles(partitions,
+        filesPerPartition);
+    return this.doWriteOperation(commitTime, operationType, newPartitionsToAdd, partitionToFilesNameLengthMap,

Review comment:
       Addressed your comment by removing the older overridden method. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749",
       "triggerID" : "947804908",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751",
       "triggerID" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2752",
       "triggerID" : "e5ba5cc8d251321d1420c2a87b9af478a517d3f4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 49f6c6bfd35edc58c95819a64815acc3beb616a5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751) 
   * e5ba5cc8d251321d1420c2a87b9af478a517d3f4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2752) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan merged pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3825:
URL: https://github.com/apache/hudi/pull/3825


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749",
       "triggerID" : "947804908",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751",
       "triggerID" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1325e87059640be4cb61f8112e7a4ccb80cd3142 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749) 
   * 49f6c6bfd35edc58c95819a64815acc3beb616a5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2751) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739",
       "triggerID" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1325e87059640be4cb61f8112e7a4ccb80cd3142",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749",
       "triggerID" : "947804908",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "49f6c6bfd35edc58c95819a64815acc3beb616a5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1325e87059640be4cb61f8112e7a4ccb80cd3142 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2739) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2749) 
   * 49f6c6bfd35edc58c95819a64815acc3beb616a5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot edited a comment on pull request #3825: [HUDI-2472] Cleaner keep latest version policy test with metadata table enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3825:
URL: https://github.com/apache/hudi/pull/3825#issuecomment-946909393


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723",
       "triggerID" : "d833f9b084264676d8ae95603d1e3c700cb493b8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d833f9b084264676d8ae95603d1e3c700cb493b8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2723) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org