You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2020/05/21 12:37:34 UTC
[GitHub] [rocketmq-externals] RongtongJin opened a new issue #568: Apache RocketMQ rebalancing architecture optimization
RongtongJin opened a new issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568
The current rebalancing process of Apache RocketMQ in cluster mode is as follows:
Each consumer instance starts a rebalance service thread, which periodically obtains a list of consumer instances under the group from the broker, and sorts the message queue under the topic and consumer Instances under the group , and then uses related strategy to calculate the message queue to be pulled. For details, please refer to this [document](https://github.com/apache/rocketmq/blob/master/docs/en/Design_LoadBlancing.md#2-consumer-load-balancing).
There are some problems with this architecture:
- In some scenarios, the view of the queue and consumer instance obtained by each consumer is inconsistent, resulting in confusion of consumer load
- The load balancing algorithm lacks stickiness, and the increase or decrease of the queue or consumer will lead to a large amount of overhead
This topic hopes to optimize the Apache RocketMQ rebalancing architecture, including
1. Move the rebalancing calculation to the broker, and the client requests the broker to obtain the allocation result.
2.Provide a sticky rebalancing algorithm for Apache RocketMQ.
3. Implementation of rebalance evaluation method in Openmessaging-Chaos.
当前Apache RocketMQ Consumer端在集群模式下的重平衡过程如下：
每个消费者实例启动一个负载均衡服务线程，该线程定时向broker获取Group下的消费者实例列表，并对Topic下的消息队列（MessageQueue）、消费者实例（Consumer id）排序，然后用分配策略算法，计算出待拉取的消息队列。详细可以参考该文档。
这样的架构存在一些问题：
• 在某些场景下，每个消费者获取到的队列和消费者实例视图不一致，导致消费者负载混乱
• 负载均衡算法缺少粘性，队列或者消费者的增减会导致大量开销
本题目希望优化Apache RocketMQ重平衡架构，主要包括
1.改进当前负载均衡架构，负载均衡的计算在broker完成，客户端请求broker获取分配结果
2.为RocketMQ提供具有粘性的负载均衡算法，在队列分尽可能平均的情况下保证和上次分配结果尽量相同
3.在OpenMessaging-Chaos中实现负载均衡评估手段。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] XiaoyiPeng edited a comment on issue #568: [KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - rebalancing re-think
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng edited a comment on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-982219231
> @LiuzcEECS, A very simple example, the consumer obtains the consumer instance list (cid list) from a randomly selected broker, but the view of cid list on different brokers may be inconsistent (the view of cid list on the broker is maintained by the consumer sending the heartbeat, the network and other reasons may cause inconsistency), inconsistency between the cid lists seen by consumers will cause confusion with rebalancing.
@RongtongJin Yeah， in kafka 0.8, rebalance is also executing on the client side, but due to Zookeeper views cannot be strictly consistent on all clients([zookeeper consistency guarantees](https://zookeeper.apache.org/doc/current/zookeeperInternals.html#sc_consistency)), kafka make rebalance on the server side from Kafka 0.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] RongtongJin commented on issue #568: [ASoC] Apache RocketMQ rebalancing architecture optimization
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-644134343
A very simple example, the consumer obtains the consumer instance list (cid list) from a randomly selected broker, but the view of cid list on different brokers may be inconsistent (the view of cid list on the broker is maintained by the consumer sending the heartbeat, the network and other reasons may cause inconsistency), inconsistency between the cid lists seen by consumers will cause confusion with rebalancing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] RongtongJin edited a comment on issue #568: [ASoC] Apache RocketMQ rebalancing architecture optimization
Posted by GitBox <gi...@apache.org>.
RongtongJin edited a comment on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-644134343
@LiuzcEECS, A very simple example, the consumer obtains the consumer instance list (cid list) from a randomly selected broker, but the view of cid list on different brokers may be inconsistent (the view of cid list on the broker is maintained by the consumer sending the heartbeat, the network and other reasons may cause inconsistency), inconsistency between the cid lists seen by consumers will cause confusion with rebalancing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] XiaoyiPeng edited a comment on issue #568: [KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - rebalancing re-think
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng edited a comment on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-982209716
I think we can take inspiration from the `Incremental Cooperative Rebalancing in Kafka`,
but in RocketMQ who plays the role of `Coordinator`(eg : `kafka.coordinator.group.GroupCoordinator`), `Broker` or `NameServer` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] XiaoyiPeng commented on issue #568: [KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - rebalancing re-think
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng commented on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-982219231
> @LiuzcEECS, A very simple example, the consumer obtains the consumer instance list (cid list) from a randomly selected broker, but the view of cid list on different brokers may be inconsistent (the view of cid list on the broker is maintained by the consumer sending the heartbeat, the network and other reasons may cause inconsistency), inconsistency between the cid lists seen by consumers will cause confusion with rebalancing.
Yeah， in kafka 0.8, rebalance is also executing on the client side, but due to Zookeeper views cannot be strictly consistent on all clients([zookeeper consistency guarantees](https://zookeeper.apache.org/doc/current/zookeeperInternals.html#sc_consistency)), kafka make rebalance on the server side from Kafka 0.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] RongtongJin removed a comment on issue #568: [SoC] Apache RocketMQ 6.0 subtasks - rebalancing architecture optimization
Posted by GitBox <gi...@apache.org>.
RongtongJin removed a comment on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-650193961
Hi, @LiuzcEECS @RonnieGandhi, ASoC's registration deadline(06.30) is approaching. If you want to participate, don't forget to submit your proposal. We look forward to your participation！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] XiaoyiPeng edited a comment on issue #568: [KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - rebalancing re-think
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng edited a comment on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-982209716
I think we can take inspiration from the `Incremental Cooperative Rebalancing in Kafka`,
but in RocketMQ who plays the role of Coodinator(eg : `kafka.coordinator.group.GroupCoordinator`), `Broker` or `NameServer` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] XiaoyiPeng removed a comment on issue #568: [KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - rebalancing re-think
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng removed a comment on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-982218676
> lly, looking forward to your participation.
Yeah， in kafka 0.8, rebalance is also executing on the client side, but due to Zookeeper views cannot be strictly consistent on all clients([zookeeper consistency guarantees](https://zookeeper.apache.org/doc/current/zookeeperInternals.html#sc_consistency)), kafka make rebalance on the server side from Kafka 0.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] XiaoyiPeng commented on issue #568: [KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - rebalancing re-think
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng commented on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-982218676
> lly, looking forward to your participation.
Yeah， in kafka 0.8, rebalance is also executing on the client side, but due to Zookeeper views cannot be strictly consistent on all clients([zookeeper consistency guarantees](https://zookeeper.apache.org/doc/current/zookeeperInternals.html#sc_consistency)), kafka make rebalance on the server side from Kafka 0.9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] RongtongJin commented on issue #568: [ASoC] Apache RocketMQ rebalancing architecture optimization
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-641697893
Hi @RonnieGandhi , Welcome to the RocketMQ community.
IMO, this is a very challenging topic, you need to have enough understanding of the code and architecture of RocketMQ. So I suggest you read the [document](https://github.com/apache/rocketmq/blob/master/docs/en/Design_LoadBlancing.md#2-consumer-load-balancing) and code about RocketMQ, especially related to consumer rebalancing. You can also refer to the implementation of other messaging platform in consumer rebalancing, such as Kafka, RabbitMQ, etc.
Finally, looking forward to your participation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] RonnieGandhi commented on issue #568: [ASoC] Apache RocketMQ rebalancing architecture optimization
Posted by GitBox <gi...@apache.org>.
RonnieGandhi commented on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-641386315
Hello sir,
I am Ronnie Gandhi, a Computer Science undergraduate student from IIT Roorkee enrolled in my pre-final year.
I am interested in working on this project. Please inform me about any pre-requisite and any specific thing needed to learn. I have previously worked on the backend architecture of an app and have some experience using the load balancer.
A brief introduction about me, I have interned at Microsoft, Hyderabad in 2019 and I am currently interning there(remotely) as well in 2020 also I have done GSoC2019 under CGAL. I am really enthusiastic about contributing to the open-source community.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] XiaoyiPeng commented on issue #568: [KEPLER-MISSION] Apache RocketMQ 6.0 subtasks - rebalancing re-think
Posted by GitBox <gi...@apache.org>.
XiaoyiPeng commented on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-982209716
I think we can take inspiration from the `Incremental Cooperative Rebalancing in Kafka`,
but who plays the role of Coodinator(eg : `kafka.coordinator.group.GroupCoordinator`), `Broker` or `NameServer` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] LiuzcEECS commented on issue #568: [ASoC] Apache RocketMQ rebalancing architecture optimization
Posted by GitBox <gi...@apache.org>.
LiuzcEECS commented on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-643870979
Hi Rongtong,
Would you please give me some examples of the scenarios where the view of the queue and consumer instance obtained by each consumer is inconsistent? Is there any issue about that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] RongtongJin removed a comment on issue #568: [SoC] Apache RocketMQ 6.0 subtasks - rebalancing architecture optimization
Posted by GitBox <gi...@apache.org>.
RongtongJin removed a comment on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-650193961
Hi, @LiuzcEECS @RonnieGandhi, ASoC's registration deadline(06.30) is approaching. If you want to participate, don't forget to submit your proposal. We look forward to your participation！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq-externals] RongtongJin commented on issue #568: [ASoC] Apache RocketMQ rebalancing architecture optimization
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on issue #568:
URL: https://github.com/apache/rocketmq-externals/issues/568#issuecomment-650193961
Hi, @LiuzcEECS @RonnieGandhi, ASoC's registration deadline(06.30) is approaching. If you want to participate, don't forget to submit your proposal. We look forward to your participation！
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org