You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/09/09 15:05:49 UTC

[GitHub] [hudi] xushiyan opened a new pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

xushiyan opened a new pull request #2079:
URL: https://github.com/apache/hudi/pull/2079


   Migrate test data prep logic in
   - TestStatsCommand
   - TestHoodieROTablePathFilter
   
   After changing to HoodieTestTable APIs, removed unused deprecated APIs in HoodieTestUtils
   
   ## *Tips*
   - *Thank you very much for contributing to Apache Hudi.*
   - *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
   
   ## What is the purpose of the pull request
   
   *(For example: This pull request adds quick-start document.)*
   
   ## Brief change log
   
   *(for example:)*
     - *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
   
   ## Verify this pull request
   
   *(Please pick either of the following options)*
   
   This pull request is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   *(example:)*
   
     - *Added integration tests for end-to-end.*
     - *Added HoodieClientWriteTest to verify the change.*
     - *Manually verified the change by running a job locally.*
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] nsivabalan commented on a change in pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #2079:
URL: https://github.com/apache/hudi/pull/2079#discussion_r488081389



##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +74,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant dateTime) {
+    return COMMIT_FORMATTER.format(Date.from(dateTime));
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {

Review comment:
       can we name this something like "makeIncrementalCommitTimes" 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] yanghua commented on a change in pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #2079:
URL: https://github.com/apache/hudi/pull/2079#discussion_r487611421



##########
File path: hudi-client/src/test/java/org/apache/hudi/table/TestCleaner.java
##########
@@ -1058,10 +1069,21 @@ public void testCleanPreviousCorruptedCleanFiles() throws IOException {
         HoodieWriteConfig.newBuilder()
             .withPath(basePath).withAssumeDatePartitioning(true)
             .withCompactionConfig(HoodieCompactionConfig.newBuilder()
-            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS).retainFileVersions(1).build())
+                .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS).retainFileVersions(1).build())

Review comment:
       redundant indent?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on a change in pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #2079:
URL: https://github.com/apache/hudi/pull/2079#discussion_r487433796



##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead

##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead

##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead

##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead

##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead

##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead

##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead

##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on a change in pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #2079:
URL: https://github.com/apache/hudi/pull/2079#discussion_r487433796



##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on a change in pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #2079:
URL: https://github.com/apache/hudi/pull/2079#discussion_r487894690



##########
File path: hudi-client/src/test/java/org/apache/hudi/table/TestCleaner.java
##########
@@ -1058,10 +1069,21 @@ public void testCleanPreviousCorruptedCleanFiles() throws IOException {
         HoodieWriteConfig.newBuilder()
             .withPath(basePath).withAssumeDatePartitioning(true)
             .withCompactionConfig(HoodieCompactionConfig.newBuilder()
-            .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS).retainFileVersions(1).build())
+                .withCleanerPolicy(HoodieCleaningPolicy.KEEP_LATEST_FILE_VERSIONS).retainFileVersions(1).build())

Review comment:
       fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on a change in pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #2079:
URL: https://github.com/apache/hudi/pull/2079#discussion_r487433796



##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +73,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant time) {
+    return COMMIT_FORMATTER.format(time);
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {
+    final Instant now = Instant.now();
+    return IntStream.range(0, num)
+        .mapToObj(i ->
+            makeNewCommitTime(now.plus(i == 0 ? firstOffsetSeconds : firstOffsetSeconds + i, SECONDS)))
+        .collect(Collectors.toList());
+  }

Review comment:
       to replace `HoodieTestUtils#monotonicIncreasingCommitTimestamps(int numTimestamps, int startSecsDelta)`

##########
File path: hudi-client/src/test/java/org/apache/hudi/testutils/HoodieClientTestHarness.java
##########
@@ -66,17 +65,12 @@
   protected transient HoodieTestDataGenerator dataGen = null;
   protected transient ExecutorService executorService;
   protected transient HoodieTableMetaClient metaClient;
-  private static AtomicInteger instantGen = new AtomicInteger(1);
   protected transient HoodieWriteClient writeClient;
   protected transient HoodieReadClient readClient;
   protected transient HoodieTableFileSystemView tableView;
 
   protected final SparkTaskContextSupplier supplier = new SparkTaskContextSupplier();
 
-  public String getNextInstant() {
-    return String.format("%09d", instantGen.getAndIncrement());
-  }
-

Review comment:
       Remove this implementation due to lack of reusability; use `HoodieTestTable#makeNewCommitTime(int sequence)` instead




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan merged pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

Posted by GitBox <gi...@apache.org>.
xushiyan merged pull request #2079:
URL: https://github.com/apache/hudi/pull/2079


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] xushiyan commented on a change in pull request #2079: [HUDI-995] Use HoodieTestTable in more classes

Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #2079:
URL: https://github.com/apache/hudi/pull/2079#discussion_r488124787



##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java
##########
@@ -66,6 +74,30 @@ public static HoodieTestTable of(HoodieTableMetaClient metaClient) {
     return new HoodieTestTable(metaClient.getBasePath(), metaClient.getRawFs(), metaClient);
   }
 
+  public static String makeNewCommitTime(int sequence) {
+    return String.format("%09d", sequence);
+  }
+
+  public static String makeNewCommitTime() {
+    return makeNewCommitTime(Instant.now());
+  }
+
+  public static String makeNewCommitTime(Instant dateTime) {
+    return COMMIT_FORMATTER.format(Date.from(dateTime));
+  }
+
+  public static List<String> makeNewCommitTimes(int num) {
+    return makeNewCommitTimes(num, 1);
+  }
+
+  public static List<String> makeNewCommitTimes(int num, int firstOffsetSeconds) {

Review comment:
       ok changed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org