[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #5269: [HUDI-3636] Create new write clients for async table services in DeltaStreamer

[GitBox <gi...@apache.org>]

alexeykudinkin commented on code in PR #5269:
URL: https://github.com/apache/hudi/pull/5269#discussion_r866299432


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java:
##########
@@ -31,16 +33,30 @@
 
   private static final long serialVersionUID = 1L;
 
+  protected final transient Object writeClientUpdateLock = new Object();
+  protected final transient List<BaseHoodieWriteClient<T, I, K, O>> oldCompactionClientList = new ArrayList<>();
+
   protected transient BaseHoodieWriteClient<T, I, K, O> compactionClient;
 
+  protected boolean isCompactionRunning = false;
+
   public BaseCompactor(BaseHoodieWriteClient<T, I, K, O> compactionClient) {
     this.compactionClient = compactionClient;
   }
 
   public abstract void compact(HoodieInstant instant) throws IOException;
 
   public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient) {
-    this.compactionClient = writeClient;
+    synchronized (writeClientUpdateLock) {
+      if (!isCompactionRunning) {
+        this.compactionClient.close();
+      } else {
+        // Store the old compaction client so that they can be closed

Review Comment:
   Agree very strongly with the point above: unless there are very strong argument why we can not re-init Async Service itself, i believe we should follow t/h with an invariant that AS and Write Client lifecycles are coupled and solve this concurrency control issues at the root



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseCompactor.java:
##########
@@ -31,16 +33,30 @@
 
   private static final long serialVersionUID = 1L;
 
+  protected final transient Object writeClientUpdateLock = new Object();
+  protected final transient List<BaseHoodieWriteClient<T, I, K, O>> oldCompactionClientList = new ArrayList<>();
+
   protected transient BaseHoodieWriteClient<T, I, K, O> compactionClient;
 
+  protected boolean isCompactionRunning = false;
+
   public BaseCompactor(BaseHoodieWriteClient<T, I, K, O> compactionClient) {
     this.compactionClient = compactionClient;
   }
 
   public abstract void compact(HoodieInstant instant) throws IOException;
 
   public void updateWriteClient(BaseHoodieWriteClient<T, I, K, O> writeClient) {
-    this.compactionClient = writeClient;
+    synchronized (writeClientUpdateLock) {
+      if (!isCompactionRunning) {
+        this.compactionClient.close();
+      } else {
+        // Store the old compaction client so that they can be closed

Review Comment:
   Immutability is hard but very powerful property we should lean in on it as much as possible, only detour from it when there's no other choice (mostly for perf reasons)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org