[GitHub] [inlong] rhizoma-atractylodis opened a new pull request, #5544: [INLONG-5101]Hash Ring

[GitBox <gi...@apache.org>]

rhizoma-atractylodis opened a new pull request, #5544:
URL: https://github.com/apache/inlong/pull/5544

   ### Motivation
   Based on my reading of the source code. Currently, the DataProxy SDK side selects DataProxy nodes using polling (sending messages in TCP mode) and random selection (sending messages in HTTP mode).The polling method is not efficient enough, and the random method is not easy to achieve load balancing.
   
   ### Changes
   Use consistent hashing algorithm instead of the original polling and random
   
   ### Mechanism Options
   Consistent Hash Algorithm and Virtual Node Mechanism
   [Refer to the article for details on the algorithm](https://blog.csdn.net/gonghaiyu/article/details/108375298)
   
   ### Design
   Based on my reading of the source code.The following are the functions that need to be modified:
   
   org.apache.inlong.sdk.dataproxy.network.ClientMgr.getClientByRoundRobin():This function obtains the DataProxy node by polling
   org.apache.inlong.sdk.dataproxy.http.InternalHttpSender.sendMessageWithHostInfo(List bodies, String groupId, String streamId, long dt, long timeout, TimeUnit timeUnit):This function implements the selection of DataProxy nodes by randomly selecting HostInfo
   Need to update the fields of the DataProxy node class to add information about virtual nodes
   The hash ring and virtual nodes need to be completed on the DataProxy side, and the strategy for acquiring DataProxy nodes on the SDK side must be updated at the same time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] baomingyu commented on a diff in pull request #5544: [INLONG-5101][DataProxy] Optimize load balancing for DataProxy

[GitBox <gi...@apache.org>]

baomingyu commented on code in PR #5544:
URL: https://github.com/apache/inlong/pull/5544#discussion_r946832977


##########
inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/network/Sender.java:
##########
@@ -513,15 +573,97 @@ private boolean validAttribute(String attr) {
     /**
      * Following methods used by asynchronously message sending.
      */
-    public void asyncSendMessage(EncodeObject encodeObject, SendMessageCallback callback, String msgUUID,
-            long timeout, TimeUnit timeUnit) throws ProxysdkException {
+    public void asyncSendMessage(EncodeObject encodeObject, SendMessageCallback callback, String msgUUID, long timeout,

Review Comment:
   
   Why not reuse next asyncSendMessage by specifying a default loadBalance type



##########
inlong-sdk/dataproxy-sdk/src/main/java/org/apache/inlong/sdk/dataproxy/utils/LoadBalance.java:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.inlong.sdk.dataproxy.utils;
+
+public class LoadBalance {

Review Comment:
   Please change it to enum type here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] rhizoma-atractylodis merged pull request #5544: [INLONG-5101][DataProxy] Optimize load balancing for DataProxy

[GitBox <gi...@apache.org>]

rhizoma-atractylodis merged PR #5544:
URL: https://github.com/apache/inlong/pull/5544


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org