You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2021/11/06 05:24:45 UTC

[GitHub] [rocketmq] lizhimins opened a new issue #3465: Support client adaptive load balancing

lizhimins opened a new issue #3465:
URL: https://github.com/apache/rocketmq/issues/3465

By default, the client uses the AllocateMessageQueueAveragely algorithm for load balancing.

As shown in the figure below, each square represents a queue on the broker side, and r and w respectively represent the actual number of read and write queues.

Under normal circumstances, the read and write traffic is balanced. When some brokers are maintained, a copy group will be in a read-only state. Although the number of read and write queues is the same as before, the actual number of writable queues is 0.

1. In the case of accumulation, all 4 clients will process the message
2. In the case of no accumulation, since new messages are only equally divided by 3 clients. Client c will be idle.
1. When we offline client c, which will cause Broker-3's queue divided equallyto other clients. It cannot be solved.
2. If there are more consumers in the subscription group or more nodes maintained by the server, the no-load situation will be more serious, with a ratio of 1-sum(writeQueue) / sum(readQueue)

Solution:

Due to RocketMQ does not currently implement an effective data source idle declaration mechanism (condition TopicRouteData Perm = 4 && MessageLag = 0), and the implementation is not simple and intuitive. A trade-off plan is to switch to the load balancing implementation of AllocateMessageQueueAveragelyByCircle, and no-load queues and ordinary read-write queues do not need to be distinguished. For subscription groups that are already online, if you use ByCircle directly, the new and old client load balancing algorithm are inconsistent, which will cause some queues in the publishing process to not consume, which will lead to accumulation

Detailed design:

1. For a brand new subscription group, it is recommended to directly use AllocateMessageQueueAveragelyByCircle for load balancing, which can significantly improve this no-load situation.

2. For groups that are already online

1. Using the default Averagely strategy, you can switch to AveragelyAuto implementation.
During load balancing, the client gets the ConsumerConnection connectionSet from the server and contains the version data numbers of all clients in the subscription group. When the client versions are all greater than a specific value, it will automatically switch to AveragelyByCircle.

2. For other strategies, no upgrade is required

Performance evaluation and compatibility:

1. Obtain all client versions MQClientAPIImpl.getConsumerConnectionList() in broker is a pure memory operation for the server
2. Client load balancing is calculated in pure memory on the client side, which has little impact on performance
3. This solution only involves the modification of the client side and does not affect the server side. Multi-language SDK can also implement the same logic

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] duhenglucky commented on issue #3465: Support client adaptive load balancing

Posted by GitBox <gi...@apache.org>.

duhenglucky commented on issue #3465:
URL: https://github.com/apache/rocketmq/issues/3465#issuecomment-962733060


   How about setting AllocateMessageQueueAveragelyByCircle as the default rebalance choice?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [rocketmq] dongeforever commented on issue #3465: Support client adaptive load balancing

Posted by GitBox <gi...@apache.org>.

dongeforever commented on issue #3465:
URL: https://github.com/apache/rocketmq/issues/3465#issuecomment-962559887


   Use the version of clients and dynamically switch to a new strategy.
   
   Good idea！Looking forward to the code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org