[GitHub] [inlong] rhizoma-atractylodis commented on issue #5101: [Feature] Optimize load balancing for DataProxy

[GitBox <gi...@apache.org>]

rhizoma-atractylodis commented on issue #5101:
URL: https://github.com/apache/inlong/issues/5101#issuecomment-1201288911

   1. The nodes joining the hash ring should be the nodes that establish the connection. After all, it is impossible to send data to the nodes that have not established the connection.
   2. When I looked at the source code before, I didn't notice that the SDK and Dataproxy interacted through the Manager, thinking they interacted directly.Then the hash ring should be done at the end that maintains the node information.Then there are two cases:（1）Manager maintains node information.The update strategy is to notify the Manager when the DataProxy node changes, and the Manager will uninstall the node in the hash ring, migrate node data according to the consistent hash algorithm, or request redirection and other operations. During this process, the node application from the SDK will be redirected to the new node, so that the client is not aware.（2）The DataProxy side maintains node data.Similar to (1), it only affects where the consistent hashing algorithm is implemented
   3. updating strategy of point 4：（1）When the DataProxy node changes, first generate virtual nodes with virtual_num number and add them to the table of real nodes and virtual nodes.（2）Add all virtual nodes to the hash ring, because the system only writes and does not read from SDK to DataProxy (if there is no misunderstanding, it should be like this). Therefore, when modifying the hash ring, if the SDK has a node request, it only needs to redirect the request to the new node.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org