答复: [External Mail][VOTE] RIP-19 RocketMQ Pop Consuming

[Xu16 Zhang 张旭 <zh...@xiaomi.com.INVALID>]

+1 approve

This feature is very useful


--------
张旭
人工智能与云平台-云技术-计算平台组
电话: 15011056041

________________________________________
发件人: heng du <du...@apache.org>
发送时间: 2021年1月18日 14:37
收件人: dev; ayanamist@gmail.com; users
主题: [External Mail][VOTE] RIP-19 RocketMQ Pop Consuming

*外部邮件，谨慎处理 | This message originated from outside of XIAOMI. Please treat this email with caution*


Hi RocketMQ Community,

This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.

In order to better implement a lightweight client, @ayanamist proposes a
new consumption model, and at the same time transfers the load balancing
logic of the original client to the broker, which not only solves the
original queue occupancy problem but also It can also avoid the consumption
delay caused by a certain consumer hangs.

The vote will be open for at least 72 hours or until a necessary number of
votes are reached.

Please vote accordingly:

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason




Best Regards!
Henry

ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：

> # RIP-19 RocketMQ Pop Consuming
>
> # Status
>
> - Current State: Proposed
> - Authors: [ayanamist]([
> https://github.com/ayanamist/](https://github.com/ayanamist/))
> - Shepherds: [duhengforever]([
> https://github.com/duhenglucky/](https://github.com/duhengforever/))
> - Mailing List discussion: dev@rocketmq.apache.org;
> users@rocketmq.apache.org
> - Pull Request: RIP-19
> - Released: -
>
> # Background & Motivation
>
> ### What do we need to do
>
> - Will we add a new module?
>
>     No.
>
> - Will we add new APIs?
>
>     Yes.
>
> - Will we add new feature?
>
>     Yes.
>
> ### Why should we do that
>
> - Are there any problems of our current project?
>
>     The current subscription load balancing strategy is based on the
> dimension of message queue. All behaviors are owned by the client side.
> There are three main steps:
>
>     1. Each consumer regularly obtains the total number of topic message
> queues and all consumers.
>     2. Using a general algorithm to sort the queues by consumer ip and
> queue index to calculate which message queue is allocated to which
> consumer.
>     3. Each consumer pulls messages using allocated orders described above.
>
>     According to this allocation method, if an abnormality occurs in a
> consumer (the application itself is abnormal, or a broker is upgrading) so
> that it causes slow subscription, messages will be accumulated, but this
> queue will not be re-allocated to another consumer, so the accumulation
> will become more and more serious.
>
>
>     Chinese version:
>
>     当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
>
>     1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
>     2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
>     3. 每个consumer去根据算法分配出来的队列，拉取消息消费
>
>
>
> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
>
> - What can we benefit proposed changes?
>
>     The accumulated messages will be subscribed by other consumers if one
> consumer behaves abnormally.
>
>     Chinese version:
>
>     在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>
> # Goals
>
> - What problem is this proposal designed to solve?
>
>     The accumulated messages will be subscribed by other consumers if one
> consumer behaves abnormally.
>
>     Chinese version:
>
>     在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>
> - To what degree should we solve the problem?
>
>     This RIP must guarantee below point:
>
>     1. High availablity: Subscription of one message queue will not be
> affected by single consumer failure.
>     2. High performance: This implementation affects latency and throughput
> less than 10%.
>
>
>     Chinese version:
>
>     新方案需要保证两点：
>
>     1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
>     2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
>
> # Non-Goals
>
> - What problem is this proposal NOT designed to solve?
>
>     Improve client-side load balancing.
>
> - Are there any limits of this proposal?
>
>     Nothing specific.
>
> # Changes
>
> ## Architecture
>
> Current "Pull mode":
> ![pull](
>
> https://user-images.githubusercontent.com/406779/103756075-cc93b900-5049-11eb-8fae-cfe5398ebaad.png
> )
>
> Proposed "Pop mode":
> ![pop](
>
> https://user-images.githubusercontent.com/406779/103757230-6d36a880-504b-11eb-95d5-7e8cff8cdef1.png
> )
>
> Move inter-queue balance of one topic from client side to server side.
> Clients make pull request without specified queues to broker, and broker
> fetch messages from queues internally and returns, which ensures one queue
> will be consumed by multiple clients. The whole behavior is like a queue
> pop process.
>
> It will add a new request command querying queue assignments in broker, and
> add pop-feature-support flag to pull request which makes broker use pop
> mode.
>
> ## Interface Design/Change
>
> - Method signature changes
>
>     Nothing specific.
>
> - Method behavior changes
>
>     Nothing specific.
>
> - CLI command changes
>
>     Add `setConsumeMode` for admin to switch between old pull mode and new
> pop mode for one subscription.
>
> - Log format or content changes
>
>     Nothing specific.
>
> ## Compatibility, Deprecation, and Migration Plan
>
> - Are backward and forward compatibility taken into consideration?
>
>     New RequestCode between client and broker are added, so there are 2
> compatibility situations:
>
>     1. old client+new broker: old clients won't make request with
> pop-feature-support flag, so broker will not enable pop mode, which keep
> all things as before.
>     2. new client+old broker: new clients will detect whether broker
> support the new request command querying queue assignments, if not, it will
> fallback to use old pull mode.
>
> - Are there deprecated APIs?
>
>     Nothing specific.
>
> - How do we do migration?
>
>     Nothing specific.
>
> ## Implementation Outline
>
> We will implement the proposed changes by **2** phases.
>
> ## Phase 1
>
> 1. Implement server-side balance capability in broker
> 2. Implement client-side request using new pop-mode
>
> ## Phase 2
>
> 1. Implement new sdk compatibility with old broker.
> 2. Implement feature detection in broker and client.
>
> # Rejected Alternatives
>
> ## How does alternatives solve the issue you proposed?
>
> Improve client rebalance logic? I don't get a quite good idea.
>
> ## Pros and Cons of alternatives
>
> Client rebalance logic will become quite complicated.
>
> ## Why should we reject above alternatives
>
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#