[GitHub] [hudi] yihua commented on a diff in pull request #7476: [HUDI-5023] Switching default Write Executor type to `SIMPLE`

[GitBox <gi...@apache.org>]

yihua commented on code in PR #7476:
URL: https://github.com/apache/hudi/pull/7476#discussion_r1060867754


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2449,12 +2450,17 @@ public Builder withWriteBufferLimitBytes(int writeBufferLimit) {
     }
 
     public Builder withWriteWaitStrategy(String waitStrategy) {
-      writeConfig.setValue(WRITE_WAIT_STRATEGY, String.valueOf(waitStrategy));
+      writeConfig.setValue(WRITE_EXECUTOR_DISRUPTOR_WAIT_STRATEGY, String.valueOf(waitStrategy));
       return this;
     }
 
     public Builder withWriteBufferSize(int size) {
-      writeConfig.setValue(WRITE_DISRUPTOR_BUFFER_SIZE, String.valueOf(size));
+      writeConfig.setValue(WRITE_EXECUTOR_DISRUPTOR_BUFFER_SIZE, String.valueOf(size));
+      return this;
+    }

Review Comment:
   nit: If this is not intended to use as a public API, could you remove this and only keep `Builder#withWriteExecutorDisruptorWriteBufferSize` which has the same functionality?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2857,8 +2863,15 @@ private void validate() {
     }
 
     public HoodieWriteConfig build() {
+      return build(true);
+    }
+
+    @VisibleForTesting
+    public HoodieWriteConfig build(boolean shouldValidate) {

Review Comment:
   should this method use the default scope/visibility instead of `public`?



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestSimpleExecutionInSpark.java:
##########
@@ -88,10 +81,10 @@ public Integer finish() {
             return count;
           }
         };
-    SimpleHoodieExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
+    SimpleExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>, Integer> exec = null;
 
     try {
-      exec = new SimpleHoodieExecutor(hoodieRecords.iterator(), consumer, getCloningTransformer(HoodieTestDataGenerator.AVRO_SCHEMA));
+      exec = new SimpleExecutor(hoodieRecords.iterator(), consumer, Function.identity());

Review Comment:
   Still use `getTransformer` instead of `Function.identity()` for the tests?



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -35,24 +36,27 @@
 import org.junit.jupiter.api.BeforeEach;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.Timeout;
+import scala.Tuple2;
 
 import java.util.ArrayList;
 import java.util.List;
 
-import scala.Tuple2;
-
-import static org.apache.hudi.execution.HoodieLazyInsertIterable.getCloningTransformer;
+import static org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformer;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.junit.jupiter.api.Assertions.assertThrows;
 import static org.junit.jupiter.api.Assertions.assertTrue;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
 
 public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
 
   private final String instantTime = HoodieActiveTimeline.createNewInstantTime();
 
+
+  private final HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder()

Review Comment:
   nit: (not required in this PR) the tests using different types of write executors should be generalized in a base class, like `TestWriteMarkersBase`, instead of duplicating the code.  This can be refactored later on.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org