You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "cchung100m (via GitHub)" <gi...@apache.org> on 2023/07/27 14:53:34 UTC

[GitHub] [incubator-uniffle] cchung100m opened a new pull request, #1046: [#363] refactor: Make the coordinator client managed by CoordinatorClientFactory singleton

cchung100m opened a new pull request, #1046:
URL: https://github.com/apache/incubator-uniffle/pull/1046

   Hi @zuston @jerqi @xianjingfeng 
   
   I am re-submitting a PR to prevent potential conflict since the last PR is been a while, any comments and suggestions would be appreciated if you are available, thank you.
   
   ### What changes were proposed in this pull request?
   
   Make the coordinator client managed by CoordinatorClientFactory singleton
   
   ### Why are the changes needed?
   
   Fix: #363
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   current UT


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] cchung100m commented on a diff in pull request #1046: [#363] refactor: Make the coordinator client managed by CoordinatorClientFactory singleton

Posted by "cchung100m (via GitHub)" <gi...@apache.org>.
cchung100m commented on code in PR #1046:
URL: https://github.com/apache/incubator-uniffle/pull/1046#discussion_r1278233147


##########
internal-client/src/main/java/org/apache/uniffle/client/factory/CoordinatorClientFactory.java:
##########
@@ -29,47 +30,68 @@
 import org.apache.uniffle.client.impl.grpc.CoordinatorGrpcClient;
 import org.apache.uniffle.common.ClientType;
 import org.apache.uniffle.common.exception.RssException;
+import org.apache.uniffle.common.util.JavaUtils;
 
 public class CoordinatorClientFactory {
   private static final Logger LOG = LoggerFactory.getLogger(CoordinatorClientFactory.class);
+  private Map<String, Map<String, CoordinatorClient>> clients = JavaUtils.newConcurrentMap();
 
-  private ClientType clientType;
+  private static class LazyHolder {
+    static final CoordinatorClientFactory INSTANCE = new CoordinatorClientFactory();
+  }
 
-  public CoordinatorClientFactory(ClientType clientType) {
-    this.clientType = clientType;
+  public static CoordinatorClientFactory getInstance() {
+    return LazyHolder.INSTANCE;
   }
 
-  public CoordinatorClient createCoordinatorClient(String host, int port) {
-    if (clientType.equals(ClientType.GRPC) || clientType.equals(ClientType.GRPC_NETTY)) {
-      return new CoordinatorGrpcClient(host, port);
-    } else {
-      throw new UnsupportedOperationException("Unsupported client type " + clientType);
+  public CoordinatorClient getOrCreateCoordinatorClient(String type, String host, int port) {
+    String coordinator = host.concat(String.valueOf(port));
+    Map<String, CoordinatorClient> typeToClients = clients.get(type);
+    if (typeToClients != null) {
+      CoordinatorClient coordinatorClient = typeToClients.get(coordinator);
+      if (coordinatorClient != null) {
+        return coordinatorClient;
+      }
+    }
+
+    switch (ClientType.valueOf(type)) {
+      case GRPC:
+        CoordinatorClient grpcClient = new CoordinatorGrpcClient(host, port);
+        typeToClients.put(coordinator, grpcClient);
+        clients.put(type, typeToClients);
+        return grpcClient;
+      default:
+        throw new UnsupportedOperationException("Unsupported client type " + type);
     }
   }
 
-  public List<CoordinatorClient> createCoordinatorClient(String coordinators) {
-    LOG.info("Start to create coordinator clients from {}", coordinators);
-    List<CoordinatorClient> coordinatorClients = Lists.newLinkedList();
+  public List<CoordinatorClient> getOrCreateCoordinatorClients(String type, String coordinators) {
     String[] coordinatorList = coordinators.trim().split(",");
     if (coordinatorList.length == 0) {
       String msg = "Invalid " + coordinators;
       LOG.error(msg);
       throw new RssException(msg);
     }
-
+    List<CoordinatorClient> coordinatorClients = Lists.newLinkedList();
     for (String coordinator : coordinatorList) {
+      clients.computeIfAbsent(type, key -> JavaUtils.newConcurrentMap());

Review Comment:
   Yes, we should leave the work to `getOrCreateCoordinatorClients`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] cchung100m closed pull request #1046: [#363] refactor: Make the coordinator client managed by CoordinatorClientFactory singleton

Posted by "cchung100m (via GitHub)" <gi...@apache.org>.
cchung100m closed pull request #1046: [#363] refactor: Make the coordinator client managed by CoordinatorClientFactory singleton
URL: https://github.com/apache/incubator-uniffle/pull/1046


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] cchung100m commented on a diff in pull request #1046: [#363] refactor: Make the coordinator client managed by CoordinatorClientFactory singleton

Posted by "cchung100m (via GitHub)" <gi...@apache.org>.
cchung100m commented on code in PR #1046:
URL: https://github.com/apache/incubator-uniffle/pull/1046#discussion_r1278232058


##########
internal-client/src/main/java/org/apache/uniffle/client/factory/CoordinatorClientFactory.java:
##########
@@ -29,47 +30,68 @@
 import org.apache.uniffle.client.impl.grpc.CoordinatorGrpcClient;
 import org.apache.uniffle.common.ClientType;
 import org.apache.uniffle.common.exception.RssException;
+import org.apache.uniffle.common.util.JavaUtils;
 
 public class CoordinatorClientFactory {
   private static final Logger LOG = LoggerFactory.getLogger(CoordinatorClientFactory.class);
+  private Map<String, Map<String, CoordinatorClient>> clients = JavaUtils.newConcurrentMap();
 
-  private ClientType clientType;
+  private static class LazyHolder {
+    static final CoordinatorClientFactory INSTANCE = new CoordinatorClientFactory();
+  }
 
-  public CoordinatorClientFactory(ClientType clientType) {
-    this.clientType = clientType;
+  public static CoordinatorClientFactory getInstance() {
+    return LazyHolder.INSTANCE;
   }
 
-  public CoordinatorClient createCoordinatorClient(String host, int port) {
-    if (clientType.equals(ClientType.GRPC) || clientType.equals(ClientType.GRPC_NETTY)) {
-      return new CoordinatorGrpcClient(host, port);
-    } else {
-      throw new UnsupportedOperationException("Unsupported client type " + clientType);
+  public CoordinatorClient getOrCreateCoordinatorClient(String type, String host, int port) {
+    String coordinator = host.concat(String.valueOf(port));
+    Map<String, CoordinatorClient> typeToClients = clients.get(type);
+    if (typeToClients != null) {
+      CoordinatorClient coordinatorClient = typeToClients.get(coordinator);
+      if (coordinatorClient != null) {
+        return coordinatorClient;
+      }
+    }
+
+    switch (ClientType.valueOf(type)) {
+      case GRPC:
+        CoordinatorClient grpcClient = new CoordinatorGrpcClient(host, port);
+        typeToClients.put(coordinator, grpcClient);

Review Comment:
   Good catch! it should be null here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] cchung100m commented on pull request #1046: [#363] refactor: Make the coordinator client managed by CoordinatorClientFactory singleton

Posted by "cchung100m (via GitHub)" <gi...@apache.org>.
cchung100m commented on PR #1046:
URL: https://github.com/apache/incubator-uniffle/pull/1046#issuecomment-1674439224

   Hi @jerqi 
   I am still working on the integration test failure. May I know what is this test actually doing?
   
   https://github.com/apache/incubator-uniffle/blob/13c71438c5fd614cdd08939d160afa722475beee/integration-test/common/src/test/java/org/apache/uniffle/test/CoordinatorAssignmentTest.java#L163
   
   Do we expect the server to change the shuffle nodes to `5`?
   https://github.com/apache/incubator-uniffle/blob/13c71438c5fd614cdd08939d160afa722475beee/integration-test/common/src/test/java/org/apache/uniffle/test/CoordinatorAssignmentTest.java#L178
   
   I am not sure why the `info.getServerToPartitionRanges().keySet().size()` returns `10` here
   https://github.com/apache/incubator-uniffle/blob/13c71438c5fd614cdd08939d160afa722475beee/integration-test/common/src/test/java/org/apache/uniffle/test/CoordinatorAssignmentTest.java#L182


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #1046: [#363] refactor: Make the coordinator client managed by CoordinatorClientFactory singleton

Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #1046:
URL: https://github.com/apache/incubator-uniffle/pull/1046#discussion_r1277017328


##########
internal-client/src/main/java/org/apache/uniffle/client/factory/CoordinatorClientFactory.java:
##########
@@ -29,47 +30,68 @@
 import org.apache.uniffle.client.impl.grpc.CoordinatorGrpcClient;
 import org.apache.uniffle.common.ClientType;
 import org.apache.uniffle.common.exception.RssException;
+import org.apache.uniffle.common.util.JavaUtils;
 
 public class CoordinatorClientFactory {
   private static final Logger LOG = LoggerFactory.getLogger(CoordinatorClientFactory.class);
+  private Map<String, Map<String, CoordinatorClient>> clients = JavaUtils.newConcurrentMap();
 
-  private ClientType clientType;
+  private static class LazyHolder {
+    static final CoordinatorClientFactory INSTANCE = new CoordinatorClientFactory();
+  }
 
-  public CoordinatorClientFactory(ClientType clientType) {
-    this.clientType = clientType;
+  public static CoordinatorClientFactory getInstance() {
+    return LazyHolder.INSTANCE;
   }
 
-  public CoordinatorClient createCoordinatorClient(String host, int port) {
-    if (clientType.equals(ClientType.GRPC) || clientType.equals(ClientType.GRPC_NETTY)) {
-      return new CoordinatorGrpcClient(host, port);
-    } else {
-      throw new UnsupportedOperationException("Unsupported client type " + clientType);
+  public CoordinatorClient getOrCreateCoordinatorClient(String type, String host, int port) {
+    String coordinator = host.concat(String.valueOf(port));
+    Map<String, CoordinatorClient> typeToClients = clients.get(type);
+    if (typeToClients != null) {
+      CoordinatorClient coordinatorClient = typeToClients.get(coordinator);
+      if (coordinatorClient != null) {
+        return coordinatorClient;
+      }
+    }
+
+    switch (ClientType.valueOf(type)) {
+      case GRPC:
+        CoordinatorClient grpcClient = new CoordinatorGrpcClient(host, port);
+        typeToClients.put(coordinator, grpcClient);

Review Comment:
   Would `typeToClients` be null?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #1046: [#363] refactor: Make the coordinator client managed by CoordinatorClientFactory singleton

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1046:
URL: https://github.com/apache/incubator-uniffle/pull/1046#issuecomment-1653807811

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1046?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#1046](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1046?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (883fa84) into [master](https://app.codecov.io/gh/apache/incubator-uniffle/commit/16942c21b9e1fad070a3ff1fa313c6f93f2bedc2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (16942c2) will **increase** coverage by `0.25%`.
   > The diff coverage is `16.12%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1046      +/-   ##
   ============================================
   + Coverage     54.33%   54.58%   +0.25%     
   + Complexity     2550     2401     -149     
   ============================================
     Files           382      345      -37     
     Lines         21739    18244    -3495     
     Branches       1803     1717      -86     
   ============================================
   - Hits          11811     9958    -1853     
   + Misses         9227     7692    -1535     
   + Partials        701      594     -107     
   ```
   
   
   | [Files Changed](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...iffle/client/factory/CoordinatorClientFactory.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW50ZXJuYWwtY2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL2NsaWVudC9mYWN0b3J5L0Nvb3JkaW5hdG9yQ2xpZW50RmFjdG9yeS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...he/uniffle/client/impl/ShuffleWriteClientImpl.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y2xpZW50L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL2NsaWVudC9pbXBsL1NodWZmbGVXcml0ZUNsaWVudEltcGwuamF2YQ==) | `35.98% <50.00%> (ø)` | |
   | [...a/org/apache/uniffle/server/RegisterHeartBeat.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9SZWdpc3RlckhlYXJ0QmVhdC5qYXZh) | `88.13% <100.00%> (+0.20%)` | :arrow_up: |
   
   ... and [39 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1046/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #1046: [#363] refactor: Make the coordinator client managed by CoordinatorClientFactory singleton

Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #1046:
URL: https://github.com/apache/incubator-uniffle/pull/1046#discussion_r1277016861


##########
internal-client/src/main/java/org/apache/uniffle/client/factory/CoordinatorClientFactory.java:
##########
@@ -29,47 +30,68 @@
 import org.apache.uniffle.client.impl.grpc.CoordinatorGrpcClient;
 import org.apache.uniffle.common.ClientType;
 import org.apache.uniffle.common.exception.RssException;
+import org.apache.uniffle.common.util.JavaUtils;
 
 public class CoordinatorClientFactory {
   private static final Logger LOG = LoggerFactory.getLogger(CoordinatorClientFactory.class);
+  private Map<String, Map<String, CoordinatorClient>> clients = JavaUtils.newConcurrentMap();
 
-  private ClientType clientType;
+  private static class LazyHolder {
+    static final CoordinatorClientFactory INSTANCE = new CoordinatorClientFactory();
+  }
 
-  public CoordinatorClientFactory(ClientType clientType) {
-    this.clientType = clientType;
+  public static CoordinatorClientFactory getInstance() {
+    return LazyHolder.INSTANCE;
   }
 
-  public CoordinatorClient createCoordinatorClient(String host, int port) {
-    if (clientType.equals(ClientType.GRPC) || clientType.equals(ClientType.GRPC_NETTY)) {
-      return new CoordinatorGrpcClient(host, port);
-    } else {
-      throw new UnsupportedOperationException("Unsupported client type " + clientType);
+  public CoordinatorClient getOrCreateCoordinatorClient(String type, String host, int port) {
+    String coordinator = host.concat(String.valueOf(port));
+    Map<String, CoordinatorClient> typeToClients = clients.get(type);
+    if (typeToClients != null) {
+      CoordinatorClient coordinatorClient = typeToClients.get(coordinator);
+      if (coordinatorClient != null) {
+        return coordinatorClient;
+      }
+    }
+
+    switch (ClientType.valueOf(type)) {
+      case GRPC:
+        CoordinatorClient grpcClient = new CoordinatorGrpcClient(host, port);
+        typeToClients.put(coordinator, grpcClient);
+        clients.put(type, typeToClients);
+        return grpcClient;
+      default:
+        throw new UnsupportedOperationException("Unsupported client type " + type);
     }
   }
 
-  public List<CoordinatorClient> createCoordinatorClient(String coordinators) {
-    LOG.info("Start to create coordinator clients from {}", coordinators);
-    List<CoordinatorClient> coordinatorClients = Lists.newLinkedList();
+  public List<CoordinatorClient> getOrCreateCoordinatorClients(String type, String coordinators) {
     String[] coordinatorList = coordinators.trim().split(",");
     if (coordinatorList.length == 0) {
       String msg = "Invalid " + coordinators;
       LOG.error(msg);
       throw new RssException(msg);
     }
-
+    List<CoordinatorClient> coordinatorClients = Lists.newLinkedList();
     for (String coordinator : coordinatorList) {
+      clients.computeIfAbsent(type, key -> JavaUtils.newConcurrentMap());

Review Comment:
   coordinatorClients.add(getOrCreateCoordinatorClient(type, coordinator))?
   I think `getOrCreateCoordinatorClients` can be more simple.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org