You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@rocketmq.apache.org by heng du <du...@apache.org> on 2021/01/18 06:37:27 UTC
[VOTE] RIP-19 RocketMQ Pop Consuming
Hi RocketMQ Community,
This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
In order to better implement a lightweight client, @ayanamist proposes a
new consumption model, and at the same time transfers the load balancing
logic of the original client to the broker, which not only solves the
original queue occupancy problem but also It can also avoid the consumption
delay caused by a certain consumer hangs.
The vote will be open for at least 72 hours or until a necessary number of
votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Best Regards!
Henry
ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
> # RIP-19 RocketMQ Pop Consuming
>
> # Status
>
> - Current State: Proposed
> - Authors: [ayanamist]([
> https://github.com/ayanamist/](https://github.com/ayanamist/))
> - Shepherds: [duhengforever]([
> https://github.com/duhenglucky/](https://github.com/duhengforever/))
> - Mailing List discussion: dev@rocketmq.apache.org;
> users@rocketmq.apache.org
> - Pull Request: RIP-19
> - Released: -
>
> # Background & Motivation
>
> ### What do we need to do
>
> - Will we add a new module?
>
> No.
>
> - Will we add new APIs?
>
> Yes.
>
> - Will we add new feature?
>
> Yes.
>
> ### Why should we do that
>
> - Are there any problems of our current project?
>
> The current subscription load balancing strategy is based on the
> dimension of message queue. All behaviors are owned by the client side.
> There are three main steps:
>
> 1. Each consumer regularly obtains the total number of topic message
> queues and all consumers.
> 2. Using a general algorithm to sort the queues by consumer ip and
> queue index to calculate which message queue is allocated to which
> consumer.
> 3. Each consumer pulls messages using allocated orders described above.
>
> According to this allocation method, if an abnormality occurs in a
> consumer (the application itself is abnormal, or a broker is upgrading) so
> that it causes slow subscription, messages will be accumulated, but this
> queue will not be re-allocated to another consumer, so the accumulation
> will become more and more serious.
>
>
> Chinese version:
>
> 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
>
> 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
> 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
> 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
>
>
>
> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
>
> - What can we benefit proposed changes?
>
> The accumulated messages will be subscribed by other consumers if one
> consumer behaves abnormally.
>
> Chinese version:
>
> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>
> # Goals
>
> - What problem is this proposal designed to solve?
>
> The accumulated messages will be subscribed by other consumers if one
> consumer behaves abnormally.
>
> Chinese version:
>
> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>
> - To what degree should we solve the problem?
>
> This RIP must guarantee below point:
>
> 1. High availablity: Subscription of one message queue will not be
> affected by single consumer failure.
> 2. High performance: This implementation affects latency and throughput
> less than 10%.
>
>
> Chinese version:
>
> 新方案需要保证两点：
>
> 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
> 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
>
> # Non-Goals
>
> - What problem is this proposal NOT designed to solve?
>
> Improve client-side load balancing.
>
> - Are there any limits of this proposal?
>
> Nothing specific.
>
> # Changes
>
> ## Architecture
>
> Current "Pull mode":
> 
>
> Proposed "Pop mode":
> 
>
> Move inter-queue balance of one topic from client side to server side.
> Clients make pull request without specified queues to broker, and broker
> fetch messages from queues internally and returns, which ensures one queue
> will be consumed by multiple clients. The whole behavior is like a queue
> pop process.
>
> It will add a new request command querying queue assignments in broker, and
> add pop-feature-support flag to pull request which makes broker use pop
> mode.
>
> ## Interface Design/Change
>
> - Method signature changes
>
> Nothing specific.
>
> - Method behavior changes
>
> Nothing specific.
>
> - CLI command changes
>
> Add `setConsumeMode` for admin to switch between old pull mode and new
> pop mode for one subscription.
>
> - Log format or content changes
>
> Nothing specific.
>
> ## Compatibility, Deprecation, and Migration Plan
>
> - Are backward and forward compatibility taken into consideration?
>
> New RequestCode between client and broker are added, so there are 2
> compatibility situations:
>
> 1. old client+new broker: old clients won't make request with
> pop-feature-support flag, so broker will not enable pop mode, which keep
> all things as before.
> 2. new client+old broker: new clients will detect whether broker
> support the new request command querying queue assignments, if not, it will
> fallback to use old pull mode.
>
> - Are there deprecated APIs?
>
> Nothing specific.
>
> - How do we do migration?
>
> Nothing specific.
>
> ## Implementation Outline
>
> We will implement the proposed changes by **2** phases.
>
> ## Phase 1
>
> 1. Implement server-side balance capability in broker
> 2. Implement client-side request using new pop-mode
>
> ## Phase 2
>
> 1. Implement new sdk compatibility with old broker.
> 2. Implement feature detection in broker and client.
>
> # Rejected Alternatives
>
> ## How does alternatives solve the issue you proposed?
>
> Improve client rebalance logic? I don't get a quite good idea.
>
> ## Pros and Cons of alternatives
>
> Client rebalance logic will become quite complicated.
>
> ## Why should we reject above alternatives
>
答复: [External Mail][VOTE] RIP-19 RocketMQ Pop Consuming
Posted by Xu16 Zhang 张旭 <zh...@xiaomi.com.INVALID>.
+1 approve
This feature is very useful
--------
张旭
人工智能与云平台-云技术-计算平台组
电话: 15011056041
________________________________________
发件人: heng du <du...@apache.org>
发送时间: 2021年1月18日 14:37
收件人: dev; ayanamist@gmail.com; users
主题: [External Mail][VOTE] RIP-19 RocketMQ Pop Consuming
*外部邮件，谨慎处理 | This message originated from outside of XIAOMI. Please treat this email with caution*
Hi RocketMQ Community,
This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
In order to better implement a lightweight client, @ayanamist proposes a
new consumption model, and at the same time transfers the load balancing
logic of the original client to the broker, which not only solves the
original queue occupancy problem but also It can also avoid the consumption
delay caused by a certain consumer hangs.
The vote will be open for at least 72 hours or until a necessary number of
votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Best Regards!
Henry
ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
> # RIP-19 RocketMQ Pop Consuming
>
> # Status
>
> - Current State: Proposed
> - Authors: [ayanamist]([
> https://github.com/ayanamist/](https://github.com/ayanamist/))
> - Shepherds: [duhengforever]([
> https://github.com/duhenglucky/](https://github.com/duhengforever/))
> - Mailing List discussion: dev@rocketmq.apache.org;
> users@rocketmq.apache.org
> - Pull Request: RIP-19
> - Released: -
>
> # Background & Motivation
>
> ### What do we need to do
>
> - Will we add a new module?
>
> No.
>
> - Will we add new APIs?
>
> Yes.
>
> - Will we add new feature?
>
> Yes.
>
> ### Why should we do that
>
> - Are there any problems of our current project?
>
> The current subscription load balancing strategy is based on the
> dimension of message queue. All behaviors are owned by the client side.
> There are three main steps:
>
> 1. Each consumer regularly obtains the total number of topic message
> queues and all consumers.
> 2. Using a general algorithm to sort the queues by consumer ip and
> queue index to calculate which message queue is allocated to which
> consumer.
> 3. Each consumer pulls messages using allocated orders described above.
>
> According to this allocation method, if an abnormality occurs in a
> consumer (the application itself is abnormal, or a broker is upgrading) so
> that it causes slow subscription, messages will be accumulated, but this
> queue will not be re-allocated to another consumer, so the accumulation
> will become more and more serious.
>
>
> Chinese version:
>
> 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
>
> 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
> 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
> 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
>
>
>
> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
>
> - What can we benefit proposed changes?
>
> The accumulated messages will be subscribed by other consumers if one
> consumer behaves abnormally.
>
> Chinese version:
>
> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>
> # Goals
>
> - What problem is this proposal designed to solve?
>
> The accumulated messages will be subscribed by other consumers if one
> consumer behaves abnormally.
>
> Chinese version:
>
> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>
> - To what degree should we solve the problem?
>
> This RIP must guarantee below point:
>
> 1. High availablity: Subscription of one message queue will not be
> affected by single consumer failure.
> 2. High performance: This implementation affects latency and throughput
> less than 10%.
>
>
> Chinese version:
>
> 新方案需要保证两点：
>
> 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
> 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
>
> # Non-Goals
>
> - What problem is this proposal NOT designed to solve?
>
> Improve client-side load balancing.
>
> - Are there any limits of this proposal?
>
> Nothing specific.
>
> # Changes
>
> ## Architecture
>
> Current "Pull mode":
> 
>
> Proposed "Pop mode":
> 
>
> Move inter-queue balance of one topic from client side to server side.
> Clients make pull request without specified queues to broker, and broker
> fetch messages from queues internally and returns, which ensures one queue
> will be consumed by multiple clients. The whole behavior is like a queue
> pop process.
>
> It will add a new request command querying queue assignments in broker, and
> add pop-feature-support flag to pull request which makes broker use pop
> mode.
>
> ## Interface Design/Change
>
> - Method signature changes
>
> Nothing specific.
>
> - Method behavior changes
>
> Nothing specific.
>
> - CLI command changes
>
> Add `setConsumeMode` for admin to switch between old pull mode and new
> pop mode for one subscription.
>
> - Log format or content changes
>
> Nothing specific.
>
> ## Compatibility, Deprecation, and Migration Plan
>
> - Are backward and forward compatibility taken into consideration?
>
> New RequestCode between client and broker are added, so there are 2
> compatibility situations:
>
> 1. old client+new broker: old clients won't make request with
> pop-feature-support flag, so broker will not enable pop mode, which keep
> all things as before.
> 2. new client+old broker: new clients will detect whether broker
> support the new request command querying queue assignments, if not, it will
> fallback to use old pull mode.
>
> - Are there deprecated APIs?
>
> Nothing specific.
>
> - How do we do migration?
>
> Nothing specific.
>
> ## Implementation Outline
>
> We will implement the proposed changes by **2** phases.
>
> ## Phase 1
>
> 1. Implement server-side balance capability in broker
> 2. Implement client-side request using new pop-mode
>
> ## Phase 2
>
> 1. Implement new sdk compatibility with old broker.
> 2. Implement feature detection in broker and client.
>
> # Rejected Alternatives
>
> ## How does alternatives solve the issue you proposed?
>
> Improve client rebalance logic? I don't get a quite good idea.
>
> ## Pros and Cons of alternatives
>
> Client rebalance logic will become quite complicated.
>
> ## Why should we reject above alternatives
>
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by ShannonDing <di...@apache.org>.
+1. This greatly improves the ease of use of the client and reduces the difficulty for developers to access the system.
On 01/19/2021 23:24，Rongtong Jin<ji...@mails.ucas.ac.cn> wrote：
+1, IMO, it is a good solution to solve some important issues of queue load balancing.
2021-01-18 14:37:27"heng du" <du...@apache.org>写道：
Hi RocketMQ Community,
This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
In order to better implement a lightweight client, @ayanamist proposes a new consumption model, and at the same time transfers the load balancing logic of the original client to the broker, which not only solves the original queue occupancy problem but also It can also avoid the consumption delay caused by a certain consumer hangs.
The vote will be open for at least 72 hours or until a necessary number of votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Best Regards!
Henry
ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
# RIP-19 RocketMQ Pop Consuming
# Status
- Current State: Proposed
- Authors: [ayanamist]([
https://github.com/ayanamist/](https://github.com/ayanamist/))
- Shepherds: [duhengforever]([
https://github.com/duhenglucky/](https://github.com/duhengforever/))
- Mailing List discussion: dev@rocketmq.apache.org;
users@rocketmq.apache.org
- Pull Request: RIP-19
- Released: -
# Background & Motivation
### What do we need to do
- Will we add a new module?
No.
- Will we add new APIs?
Yes.
- Will we add new feature?
Yes.
### Why should we do that
- Are there any problems of our current project?
The current subscription load balancing strategy is based on the
dimension of message queue. All behaviors are owned by the client side.
There are three main steps:
1. Each consumer regularly obtains the total number of topic message
queues and all consumers.
2. Using a general algorithm to sort the queues by consumer ip and
queue index to calculate which message queue is allocated to which consumer.
3. Each consumer pulls messages using allocated orders described above.
According to this allocation method, if an abnormality occurs in a
consumer (the application itself is abnormal, or a broker is upgrading) so
that it causes slow subscription, messages will be accumulated, but this
queue will not be re-allocated to another consumer, so the accumulation
will become more and more serious.
Chinese version:
当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
3. 每个consumer去根据算法分配出来的队列，拉取消息消费
按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
- What can we benefit proposed changes?
The accumulated messages will be subscribed by other consumers if one
consumer behaves abnormally.
Chinese version:
在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
# Goals
- What problem is this proposal designed to solve?
The accumulated messages will be subscribed by other consumers if one
consumer behaves abnormally.
Chinese version:
在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
- To what degree should we solve the problem?
This RIP must guarantee below point:
1. High availablity: Subscription of one message queue will not be
affected by single consumer failure.
2. High performance: This implementation affects latency and throughput
less than 10%.
Chinese version:
新方案需要保证两点：
1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
# Non-Goals
- What problem is this proposal NOT designed to solve?
Improve client-side load balancing.
- Are there any limits of this proposal?
Nothing specific.
# Changes
## Architecture
Current "Pull mode":
Proposed "Pop mode":
Move inter-queue balance of one topic from client side to server side.
Clients make pull request without specified queues to broker, and broker
fetch messages from queues internally and returns, which ensures one queue
will be consumed by multiple clients. The whole behavior is like a queue
pop process.
It will add a new request command querying queue assignments in broker, and
add pop-feature-support flag to pull request which makes broker use pop
mode.
## Interface Design/Change
- Method signature changes
Nothing specific.
- Method behavior changes
Nothing specific.
- CLI command changes
Add `setConsumeMode` for admin to switch between old pull mode and new
pop mode for one subscription.
- Log format or content changes
Nothing specific.
## Compatibility, Deprecation, and Migration Plan
- Are backward and forward compatibility taken into consideration?
New RequestCode between client and broker are added, so there are 2
compatibility situations:
1. old client+new broker: old clients won't make request with
pop-feature-support flag, so broker will not enable pop mode, which keep
all things as before.
2. new client+old broker: new clients will detect whether broker
support the new request command querying queue assignments, if not, it will
fallback to use old pull mode.
- Are there deprecated APIs?
Nothing specific.
- How do we do migration?
Nothing specific.
## Implementation Outline
We will implement the proposed changes by **2** phases.
## Phase 1
1. Implement server-side balance capability in broker
2. Implement client-side request using new pop-mode
## Phase 2
1. Implement new sdk compatibility with old broker.
2. Implement feature detection in broker and client.
# Rejected Alternatives
## How does alternatives solve the issue you proposed?
Improve client rebalance logic? I don't get a quite good idea.
## Pros and Cons of alternatives
Client rebalance logic will become quite complicated.
## Why should we reject above alternatives
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by ShannonDing <di...@apache.org>.
+1. This greatly improves the ease of use of the client and reduces the difficulty for developers to access the system.
On 01/19/2021 23:24，Rongtong Jin<ji...@mails.ucas.ac.cn> wrote：
+1, IMO, it is a good solution to solve some important issues of queue load balancing.
2021-01-18 14:37:27"heng du" <du...@apache.org>写道：
Hi RocketMQ Community,
This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
In order to better implement a lightweight client, @ayanamist proposes a new consumption model, and at the same time transfers the load balancing logic of the original client to the broker, which not only solves the original queue occupancy problem but also It can also avoid the consumption delay caused by a certain consumer hangs.
The vote will be open for at least 72 hours or until a necessary number of votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Best Regards!
Henry
ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
# RIP-19 RocketMQ Pop Consuming
# Status
- Current State: Proposed
- Authors: [ayanamist]([
https://github.com/ayanamist/](https://github.com/ayanamist/))
- Shepherds: [duhengforever]([
https://github.com/duhenglucky/](https://github.com/duhengforever/))
- Mailing List discussion: dev@rocketmq.apache.org;
users@rocketmq.apache.org
- Pull Request: RIP-19
- Released: -
# Background & Motivation
### What do we need to do
- Will we add a new module?
No.
- Will we add new APIs?
Yes.
- Will we add new feature?
Yes.
### Why should we do that
- Are there any problems of our current project?
The current subscription load balancing strategy is based on the
dimension of message queue. All behaviors are owned by the client side.
There are three main steps:
1. Each consumer regularly obtains the total number of topic message
queues and all consumers.
2. Using a general algorithm to sort the queues by consumer ip and
queue index to calculate which message queue is allocated to which consumer.
3. Each consumer pulls messages using allocated orders described above.
According to this allocation method, if an abnormality occurs in a
consumer (the application itself is abnormal, or a broker is upgrading) so
that it causes slow subscription, messages will be accumulated, but this
queue will not be re-allocated to another consumer, so the accumulation
will become more and more serious.
Chinese version:
当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
3. 每个consumer去根据算法分配出来的队列，拉取消息消费
按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
- What can we benefit proposed changes?
The accumulated messages will be subscribed by other consumers if one
consumer behaves abnormally.
Chinese version:
在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
# Goals
- What problem is this proposal designed to solve?
The accumulated messages will be subscribed by other consumers if one
consumer behaves abnormally.
Chinese version:
在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
- To what degree should we solve the problem?
This RIP must guarantee below point:
1. High availablity: Subscription of one message queue will not be
affected by single consumer failure.
2. High performance: This implementation affects latency and throughput
less than 10%.
Chinese version:
新方案需要保证两点：
1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
# Non-Goals
- What problem is this proposal NOT designed to solve?
Improve client-side load balancing.
- Are there any limits of this proposal?
Nothing specific.
# Changes
## Architecture
Current "Pull mode":
Proposed "Pop mode":
Move inter-queue balance of one topic from client side to server side.
Clients make pull request without specified queues to broker, and broker
fetch messages from queues internally and returns, which ensures one queue
will be consumed by multiple clients. The whole behavior is like a queue
pop process.
It will add a new request command querying queue assignments in broker, and
add pop-feature-support flag to pull request which makes broker use pop
mode.
## Interface Design/Change
- Method signature changes
Nothing specific.
- Method behavior changes
Nothing specific.
- CLI command changes
Add `setConsumeMode` for admin to switch between old pull mode and new
pop mode for one subscription.
- Log format or content changes
Nothing specific.
## Compatibility, Deprecation, and Migration Plan
- Are backward and forward compatibility taken into consideration?
New RequestCode between client and broker are added, so there are 2
compatibility situations:
1. old client+new broker: old clients won't make request with
pop-feature-support flag, so broker will not enable pop mode, which keep
all things as before.
2. new client+old broker: new clients will detect whether broker
support the new request command querying queue assignments, if not, it will
fallback to use old pull mode.
- Are there deprecated APIs?
Nothing specific.
- How do we do migration?
Nothing specific.
## Implementation Outline
We will implement the proposed changes by **2** phases.
## Phase 1
1. Implement server-side balance capability in broker
2. Implement client-side request using new pop-mode
## Phase 2
1. Implement new sdk compatibility with old broker.
2. Implement feature detection in broker and client.
# Rejected Alternatives
## How does alternatives solve the issue you proposed?
Improve client rebalance logic? I don't get a quite good idea.
## Pros and Cons of alternatives
Client rebalance logic will become quite complicated.
## Why should we reject above alternatives
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by Rongtong Jin <ji...@mails.ucas.ac.cn>.
+1, IMO, it is a good solution to solve some important issues of queue load balancing.
2021-01-18 14:37:27"heng du" <du...@apache.org>写道：
Hi RocketMQ Community,
This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
In order to better implement a lightweight client, @ayanamist proposes a new consumption model, and at the same time transfers the load balancing logic of the original client to the broker, which not only solves the original queue occupancy problem but also It can also avoid the consumption delay caused by a certain consumer hangs.
The vote will be open for at least 72 hours or until a necessary number of votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Best Regards!
Henry
ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
# RIP-19 RocketMQ Pop Consuming
# Status
- Current State: Proposed
- Authors: [ayanamist]([
https://github.com/ayanamist/](https://github.com/ayanamist/))
- Shepherds: [duhengforever]([
https://github.com/duhenglucky/](https://github.com/duhengforever/))
- Mailing List discussion: dev@rocketmq.apache.org;
users@rocketmq.apache.org
- Pull Request: RIP-19
- Released: -
# Background & Motivation
### What do we need to do
- Will we add a new module?
No.
- Will we add new APIs?
Yes.
- Will we add new feature?
Yes.
### Why should we do that
- Are there any problems of our current project?
The current subscription load balancing strategy is based on the
dimension of message queue. All behaviors are owned by the client side.
There are three main steps:
1. Each consumer regularly obtains the total number of topic message
queues and all consumers.
2. Using a general algorithm to sort the queues by consumer ip and
queue index to calculate which message queue is allocated to which consumer.
3. Each consumer pulls messages using allocated orders described above.
According to this allocation method, if an abnormality occurs in a
consumer (the application itself is abnormal, or a broker is upgrading) so
that it causes slow subscription, messages will be accumulated, but this
queue will not be re-allocated to another consumer, so the accumulation
will become more and more serious.
Chinese version:
当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
3. 每个consumer去根据算法分配出来的队列，拉取消息消费
按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
- What can we benefit proposed changes?
The accumulated messages will be subscribed by other consumers if one
consumer behaves abnormally.
Chinese version:
在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
# Goals
- What problem is this proposal designed to solve?
The accumulated messages will be subscribed by other consumers if one
consumer behaves abnormally.
Chinese version:
在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
- To what degree should we solve the problem?
This RIP must guarantee below point:
1. High availablity: Subscription of one message queue will not be
affected by single consumer failure.
2. High performance: This implementation affects latency and throughput
less than 10%.
Chinese version:
新方案需要保证两点：
1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
# Non-Goals
- What problem is this proposal NOT designed to solve?
Improve client-side load balancing.
- Are there any limits of this proposal?
Nothing specific.
# Changes
## Architecture
Current "Pull mode":
Proposed "Pop mode":
Move inter-queue balance of one topic from client side to server side.
Clients make pull request without specified queues to broker, and broker
fetch messages from queues internally and returns, which ensures one queue
will be consumed by multiple clients. The whole behavior is like a queue
pop process.
It will add a new request command querying queue assignments in broker, and
add pop-feature-support flag to pull request which makes broker use pop
mode.
## Interface Design/Change
- Method signature changes
Nothing specific.
- Method behavior changes
Nothing specific.
- CLI command changes
Add `setConsumeMode` for admin to switch between old pull mode and new
pop mode for one subscription.
- Log format or content changes
Nothing specific.
## Compatibility, Deprecation, and Migration Plan
- Are backward and forward compatibility taken into consideration?
New RequestCode between client and broker are added, so there are 2
compatibility situations:
1. old client+new broker: old clients won't make request with
pop-feature-support flag, so broker will not enable pop mode, which keep
all things as before.
2. new client+old broker: new clients will detect whether broker
support the new request command querying queue assignments, if not, it will
fallback to use old pull mode.
- Are there deprecated APIs?
Nothing specific.
- How do we do migration?
Nothing specific.
## Implementation Outline
We will implement the proposed changes by **2** phases.
## Phase 1
1. Implement server-side balance capability in broker
2. Implement client-side request using new pop-mode
## Phase 2
1. Implement new sdk compatibility with old broker.
2. Implement feature detection in broker and client.
# Rejected Alternatives
## How does alternatives solve the issue you proposed?
Improve client rebalance logic? I don't get a quite good idea.
## Pros and Cons of alternatives
Client rebalance logic will become quite complicated.
## Why should we reject above alternatives
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by liqipeng <wl...@163.com>.
+1
> 在 2021年1月18日，下午2:37，heng du <du...@apache.org> 写道：
>
> Hi RocketMQ Community,
>
> This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
>
> In order to better implement a lightweight client, @ayanamist proposes a
> new consumption model, and at the same time transfers the load balancing
> logic of the original client to the broker, which not only solves the
> original queue occupancy problem but also It can also avoid the consumption
> delay caused by a certain consumer hangs.
>
> The vote will be open for at least 72 hours or until a necessary number of
> votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
>
>
> Best Regards!
> Henry
>
> ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
>
>> # RIP-19 RocketMQ Pop Consuming
>>
>> # Status
>>
>> - Current State: Proposed
>> - Authors: [ayanamist]([
>> https://github.com/ayanamist/](https://github.com/ayanamist/))
>> - Shepherds: [duhengforever]([
>> https://github.com/duhenglucky/](https://github.com/duhengforever/))
>> - Mailing List discussion: dev@rocketmq.apache.org;
>> users@rocketmq.apache.org
>> - Pull Request: RIP-19
>> - Released: -
>>
>> # Background & Motivation
>>
>> ### What do we need to do
>>
>> - Will we add a new module?
>>
>> No.
>>
>> - Will we add new APIs?
>>
>> Yes.
>>
>> - Will we add new feature?
>>
>> Yes.
>>
>> ### Why should we do that
>>
>> - Are there any problems of our current project?
>>
>> The current subscription load balancing strategy is based on the
>> dimension of message queue. All behaviors are owned by the client side.
>> There are three main steps:
>>
>> 1. Each consumer regularly obtains the total number of topic message
>> queues and all consumers.
>> 2. Using a general algorithm to sort the queues by consumer ip and
>> queue index to calculate which message queue is allocated to which
>> consumer.
>> 3. Each consumer pulls messages using allocated orders described above.
>>
>> According to this allocation method, if an abnormality occurs in a
>> consumer (the application itself is abnormal, or a broker is upgrading) so
>> that it causes slow subscription, messages will be accumulated, but this
>> queue will not be re-allocated to another consumer, so the accumulation
>> will become more and more serious.
>>
>>
>> Chinese version:
>>
>> 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
>>
>> 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
>> 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
>> 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
>>
>>
>>
>> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
>>
>> - What can we benefit proposed changes?
>>
>> The accumulated messages will be subscribed by other consumers if one
>> consumer behaves abnormally.
>>
>> Chinese version:
>>
>> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>>
>> # Goals
>>
>> - What problem is this proposal designed to solve?
>>
>> The accumulated messages will be subscribed by other consumers if one
>> consumer behaves abnormally.
>>
>> Chinese version:
>>
>> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>>
>> - To what degree should we solve the problem?
>>
>> This RIP must guarantee below point:
>>
>> 1. High availablity: Subscription of one message queue will not be
>> affected by single consumer failure.
>> 2. High performance: This implementation affects latency and throughput
>> less than 10%.
>>
>>
>> Chinese version:
>>
>> 新方案需要保证两点：
>>
>> 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
>> 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
>>
>> # Non-Goals
>>
>> - What problem is this proposal NOT designed to solve?
>>
>> Improve client-side load balancing.
>>
>> - Are there any limits of this proposal?
>>
>> Nothing specific.
>>
>> # Changes
>>
>> ## Architecture
>>
>> Current "Pull mode":
>> 
>>
>> Proposed "Pop mode":
>> 
>>
>> Move inter-queue balance of one topic from client side to server side.
>> Clients make pull request without specified queues to broker, and broker
>> fetch messages from queues internally and returns, which ensures one queue
>> will be consumed by multiple clients. The whole behavior is like a queue
>> pop process.
>>
>> It will add a new request command querying queue assignments in broker, and
>> add pop-feature-support flag to pull request which makes broker use pop
>> mode.
>>
>> ## Interface Design/Change
>>
>> - Method signature changes
>>
>> Nothing specific.
>>
>> - Method behavior changes
>>
>> Nothing specific.
>>
>> - CLI command changes
>>
>> Add `setConsumeMode` for admin to switch between old pull mode and new
>> pop mode for one subscription.
>>
>> - Log format or content changes
>>
>> Nothing specific.
>>
>> ## Compatibility, Deprecation, and Migration Plan
>>
>> - Are backward and forward compatibility taken into consideration?
>>
>> New RequestCode between client and broker are added, so there are 2
>> compatibility situations:
>>
>> 1. old client+new broker: old clients won't make request with
>> pop-feature-support flag, so broker will not enable pop mode, which keep
>> all things as before.
>> 2. new client+old broker: new clients will detect whether broker
>> support the new request command querying queue assignments, if not, it will
>> fallback to use old pull mode.
>>
>> - Are there deprecated APIs?
>>
>> Nothing specific.
>>
>> - How do we do migration?
>>
>> Nothing specific.
>>
>> ## Implementation Outline
>>
>> We will implement the proposed changes by **2** phases.
>>
>> ## Phase 1
>>
>> 1. Implement server-side balance capability in broker
>> 2. Implement client-side request using new pop-mode
>>
>> ## Phase 2
>>
>> 1. Implement new sdk compatibility with old broker.
>> 2. Implement feature detection in broker and client.
>>
>> # Rejected Alternatives
>>
>> ## How does alternatives solve the issue you proposed?
>>
>> Improve client rebalance logic? I don't get a quite good idea.
>>
>> ## Pros and Cons of alternatives
>>
>> Client rebalance logic will become quite complicated.
>>
>> ## Why should we reject above alternatives
>>
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by vongosling <fe...@gmail.com>.
+1,
This is a correct and great step for the cloud-native infra, especially for
RocketMQ community. Looking forward to seeing a more practical and concrete
proposal.
panwei zhang <zh...@gmail.com> 于2021年1月20日周三 下午9:59写道：
> +1
>
> tiger lee <ti...@apache.org> 于2021年1月20日周三 下午3:32写道：
>
>> [ ] +1 approve
>> it would make Consumer Client more simple to use. and do u have a work
>> flow to show how POP mode work ?
>>
>> heng du <du...@apache.org> 于2021年1月18日周一 下午2:37写道：
>>
>>> Hi RocketMQ Community,
>>>
>>> This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
>>>
>>> In order to better implement a lightweight client, @ayanamist proposes a
>>> new consumption model, and at the same time transfers the load balancing
>>> logic of the original client to the broker, which not only solves the
>>> original queue occupancy problem but also It can also avoid the
>>> consumption
>>> delay caused by a certain consumer hangs.
>>>
>>> The vote will be open for at least 72 hours or until a necessary number
>>> of
>>> votes are reached.
>>>
>>> Please vote accordingly:
>>>
>>> [ ] +1 approve
>>> [ ] +0 no opinion
>>> [ ] -1 disapprove with the reason
>>>
>>>
>>>
>>>
>>> Best Regards!
>>> Henry
>>>
>>> ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
>>>
>>> > # RIP-19 RocketMQ Pop Consuming
>>> >
>>> > # Status
>>> >
>>> > - Current State: Proposed
>>> > - Authors: [ayanamist]([
>>> > https://github.com/ayanamist/](https://github.com/ayanamist/))
>>> > - Shepherds: [duhengforever]([
>>> > https://github.com/duhenglucky/](https://github.com/duhengforever/))
>>> > - Mailing List discussion: dev@rocketmq.apache.org;
>>> > users@rocketmq.apache.org
>>> > - Pull Request: RIP-19
>>> > - Released: -
>>> >
>>> > # Background & Motivation
>>> >
>>> > ### What do we need to do
>>> >
>>> > - Will we add a new module?
>>> >
>>> > No.
>>> >
>>> > - Will we add new APIs?
>>> >
>>> > Yes.
>>> >
>>> > - Will we add new feature?
>>> >
>>> > Yes.
>>> >
>>> > ### Why should we do that
>>> >
>>> > - Are there any problems of our current project?
>>> >
>>> > The current subscription load balancing strategy is based on the
>>> > dimension of message queue. All behaviors are owned by the client side.
>>> > There are three main steps:
>>> >
>>> > 1. Each consumer regularly obtains the total number of topic
>>> message
>>> > queues and all consumers.
>>> > 2. Using a general algorithm to sort the queues by consumer ip and
>>> > queue index to calculate which message queue is allocated to which
>>> > consumer.
>>> > 3. Each consumer pulls messages using allocated orders described
>>> above.
>>> >
>>> > According to this allocation method, if an abnormality occurs in a
>>> > consumer (the application itself is abnormal, or a broker is
>>> upgrading) so
>>> > that it causes slow subscription, messages will be accumulated, but
>>> this
>>> > queue will not be re-allocated to another consumer, so the accumulation
>>> > will become more and more serious.
>>> >
>>> >
>>> > Chinese version:
>>> >
>>> > 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
>>> >
>>> > 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
>>> > 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
>>> > 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
>>> >
>>> >
>>> >
>>> >
>>> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
>>> >
>>> > - What can we benefit proposed changes?
>>> >
>>> > The accumulated messages will be subscribed by other consumers if
>>> one
>>> > consumer behaves abnormally.
>>> >
>>> > Chinese version:
>>> >
>>> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>>> >
>>> > # Goals
>>> >
>>> > - What problem is this proposal designed to solve?
>>> >
>>> > The accumulated messages will be subscribed by other consumers if
>>> one
>>> > consumer behaves abnormally.
>>> >
>>> > Chinese version:
>>> >
>>> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>>> >
>>> > - To what degree should we solve the problem?
>>> >
>>> > This RIP must guarantee below point:
>>> >
>>> > 1. High availablity: Subscription of one message queue will not be
>>> > affected by single consumer failure.
>>> > 2. High performance: This implementation affects latency and
>>> throughput
>>> > less than 10%.
>>> >
>>> >
>>> > Chinese version:
>>> >
>>> > 新方案需要保证两点：
>>> >
>>> > 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
>>> > 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
>>> >
>>> > # Non-Goals
>>> >
>>> > - What problem is this proposal NOT designed to solve?
>>> >
>>> > Improve client-side load balancing.
>>> >
>>> > - Are there any limits of this proposal?
>>> >
>>> > Nothing specific.
>>> >
>>> > # Changes
>>> >
>>> > ## Architecture
>>> >
>>> > Current "Pull mode":
>>> > 
>>> >
>>> > Proposed "Pop mode":
>>> > 
>>> >
>>> > Move inter-queue balance of one topic from client side to server side.
>>> > Clients make pull request without specified queues to broker, and
>>> broker
>>> > fetch messages from queues internally and returns, which ensures one
>>> queue
>>> > will be consumed by multiple clients. The whole behavior is like a
>>> queue
>>> > pop process.
>>> >
>>> > It will add a new request command querying queue assignments in
>>> broker, and
>>> > add pop-feature-support flag to pull request which makes broker use pop
>>> > mode.
>>> >
>>> > ## Interface Design/Change
>>> >
>>> > - Method signature changes
>>> >
>>> > Nothing specific.
>>> >
>>> > - Method behavior changes
>>> >
>>> > Nothing specific.
>>> >
>>> > - CLI command changes
>>> >
>>> > Add `setConsumeMode` for admin to switch between old pull mode and
>>> new
>>> > pop mode for one subscription.
>>> >
>>> > - Log format or content changes
>>> >
>>> > Nothing specific.
>>> >
>>> > ## Compatibility, Deprecation, and Migration Plan
>>> >
>>> > - Are backward and forward compatibility taken into consideration?
>>> >
>>> > New RequestCode between client and broker are added, so there are 2
>>> > compatibility situations:
>>> >
>>> > 1. old client+new broker: old clients won't make request with
>>> > pop-feature-support flag, so broker will not enable pop mode, which
>>> keep
>>> > all things as before.
>>> > 2. new client+old broker: new clients will detect whether broker
>>> > support the new request command querying queue assignments, if not, it
>>> will
>>> > fallback to use old pull mode.
>>> >
>>> > - Are there deprecated APIs?
>>> >
>>> > Nothing specific.
>>> >
>>> > - How do we do migration?
>>> >
>>> > Nothing specific.
>>> >
>>> > ## Implementation Outline
>>> >
>>> > We will implement the proposed changes by **2** phases.
>>> >
>>> > ## Phase 1
>>> >
>>> > 1. Implement server-side balance capability in broker
>>> > 2. Implement client-side request using new pop-mode
>>> >
>>> > ## Phase 2
>>> >
>>> > 1. Implement new sdk compatibility with old broker.
>>> > 2. Implement feature detection in broker and client.
>>> >
>>> > # Rejected Alternatives
>>> >
>>> > ## How does alternatives solve the issue you proposed?
>>> >
>>> > Improve client rebalance logic? I don't get a quite good idea.
>>> >
>>> > ## Pros and Cons of alternatives
>>> >
>>> > Client rebalance logic will become quite complicated.
>>> >
>>> > ## Why should we reject above alternatives
>>> >
>>>
>>
--
Nothing is impossible
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by panwei zhang <zh...@gmail.com>.
+1
tiger lee <ti...@apache.org> 于2021年1月20日周三 下午3:32写道：
> [ ] +1 approve
> it would make Consumer Client more simple to use. and do u have a work
> flow to show how POP mode work ?
>
> heng du <du...@apache.org> 于2021年1月18日周一 下午2:37写道：
>
>> Hi RocketMQ Community,
>>
>> This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
>>
>> In order to better implement a lightweight client, @ayanamist proposes a
>> new consumption model, and at the same time transfers the load balancing
>> logic of the original client to the broker, which not only solves the
>> original queue occupancy problem but also It can also avoid the
>> consumption
>> delay caused by a certain consumer hangs.
>>
>> The vote will be open for at least 72 hours or until a necessary number of
>> votes are reached.
>>
>> Please vote accordingly:
>>
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>>
>>
>>
>>
>> Best Regards!
>> Henry
>>
>> ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
>>
>> > # RIP-19 RocketMQ Pop Consuming
>> >
>> > # Status
>> >
>> > - Current State: Proposed
>> > - Authors: [ayanamist]([
>> > https://github.com/ayanamist/](https://github.com/ayanamist/))
>> > - Shepherds: [duhengforever]([
>> > https://github.com/duhenglucky/](https://github.com/duhengforever/))
>> > - Mailing List discussion: dev@rocketmq.apache.org;
>> > users@rocketmq.apache.org
>> > - Pull Request: RIP-19
>> > - Released: -
>> >
>> > # Background & Motivation
>> >
>> > ### What do we need to do
>> >
>> > - Will we add a new module?
>> >
>> > No.
>> >
>> > - Will we add new APIs?
>> >
>> > Yes.
>> >
>> > - Will we add new feature?
>> >
>> > Yes.
>> >
>> > ### Why should we do that
>> >
>> > - Are there any problems of our current project?
>> >
>> > The current subscription load balancing strategy is based on the
>> > dimension of message queue. All behaviors are owned by the client side.
>> > There are three main steps:
>> >
>> > 1. Each consumer regularly obtains the total number of topic message
>> > queues and all consumers.
>> > 2. Using a general algorithm to sort the queues by consumer ip and
>> > queue index to calculate which message queue is allocated to which
>> > consumer.
>> > 3. Each consumer pulls messages using allocated orders described
>> above.
>> >
>> > According to this allocation method, if an abnormality occurs in a
>> > consumer (the application itself is abnormal, or a broker is upgrading)
>> so
>> > that it causes slow subscription, messages will be accumulated, but this
>> > queue will not be re-allocated to another consumer, so the accumulation
>> > will become more and more serious.
>> >
>> >
>> > Chinese version:
>> >
>> > 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
>> >
>> > 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
>> > 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
>> > 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
>> >
>> >
>> >
>> >
>> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
>> >
>> > - What can we benefit proposed changes?
>> >
>> > The accumulated messages will be subscribed by other consumers if
>> one
>> > consumer behaves abnormally.
>> >
>> > Chinese version:
>> >
>> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>> >
>> > # Goals
>> >
>> > - What problem is this proposal designed to solve?
>> >
>> > The accumulated messages will be subscribed by other consumers if
>> one
>> > consumer behaves abnormally.
>> >
>> > Chinese version:
>> >
>> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>> >
>> > - To what degree should we solve the problem?
>> >
>> > This RIP must guarantee below point:
>> >
>> > 1. High availablity: Subscription of one message queue will not be
>> > affected by single consumer failure.
>> > 2. High performance: This implementation affects latency and
>> throughput
>> > less than 10%.
>> >
>> >
>> > Chinese version:
>> >
>> > 新方案需要保证两点：
>> >
>> > 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
>> > 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
>> >
>> > # Non-Goals
>> >
>> > - What problem is this proposal NOT designed to solve?
>> >
>> > Improve client-side load balancing.
>> >
>> > - Are there any limits of this proposal?
>> >
>> > Nothing specific.
>> >
>> > # Changes
>> >
>> > ## Architecture
>> >
>> > Current "Pull mode":
>> > 
>> >
>> > Proposed "Pop mode":
>> > 
>> >
>> > Move inter-queue balance of one topic from client side to server side.
>> > Clients make pull request without specified queues to broker, and broker
>> > fetch messages from queues internally and returns, which ensures one
>> queue
>> > will be consumed by multiple clients. The whole behavior is like a queue
>> > pop process.
>> >
>> > It will add a new request command querying queue assignments in broker,
>> and
>> > add pop-feature-support flag to pull request which makes broker use pop
>> > mode.
>> >
>> > ## Interface Design/Change
>> >
>> > - Method signature changes
>> >
>> > Nothing specific.
>> >
>> > - Method behavior changes
>> >
>> > Nothing specific.
>> >
>> > - CLI command changes
>> >
>> > Add `setConsumeMode` for admin to switch between old pull mode and
>> new
>> > pop mode for one subscription.
>> >
>> > - Log format or content changes
>> >
>> > Nothing specific.
>> >
>> > ## Compatibility, Deprecation, and Migration Plan
>> >
>> > - Are backward and forward compatibility taken into consideration?
>> >
>> > New RequestCode between client and broker are added, so there are 2
>> > compatibility situations:
>> >
>> > 1. old client+new broker: old clients won't make request with
>> > pop-feature-support flag, so broker will not enable pop mode, which keep
>> > all things as before.
>> > 2. new client+old broker: new clients will detect whether broker
>> > support the new request command querying queue assignments, if not, it
>> will
>> > fallback to use old pull mode.
>> >
>> > - Are there deprecated APIs?
>> >
>> > Nothing specific.
>> >
>> > - How do we do migration?
>> >
>> > Nothing specific.
>> >
>> > ## Implementation Outline
>> >
>> > We will implement the proposed changes by **2** phases.
>> >
>> > ## Phase 1
>> >
>> > 1. Implement server-side balance capability in broker
>> > 2. Implement client-side request using new pop-mode
>> >
>> > ## Phase 2
>> >
>> > 1. Implement new sdk compatibility with old broker.
>> > 2. Implement feature detection in broker and client.
>> >
>> > # Rejected Alternatives
>> >
>> > ## How does alternatives solve the issue you proposed?
>> >
>> > Improve client rebalance logic? I don't get a quite good idea.
>> >
>> > ## Pros and Cons of alternatives
>> >
>> > Client rebalance logic will become quite complicated.
>> >
>> > ## Why should we reject above alternatives
>> >
>>
>
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by vongosling <fe...@gmail.com>.
Pop is not a appropriate terminology, I would suggest a more general term.
such as lpull(lite pull)? The previous pull model will be retained. About
this, I'd like to hear more voices from the community.
ayanamist <ay...@gmail.com> 于2021年1月20日周三 下午5:09写道：
> I will supplement a work flow later.
> Internally, it adds "POP"+"ACK" requests to broker+client interaction which
> "POP" fetches messages and "ACK" acknowledges these messages' offsets, then
> all flow can be moved from client to broker side. There will be some
> changes in push consumer so that it keeps client usage as same as before,
> and broker controls whether this consumer group use POP or PULL
>
> On Wed, Jan 20, 2021 at 3:32 PM tiger lee <ti...@apache.org> wrote:
>
> > [ ] +1 approve
> > it would make Consumer Client more simple to use. and do u have a work
> > flow to show how POP mode work ?
> >
> > heng du <du...@apache.org> 于2021年1月18日周一 下午2:37写道：
> >
> > > Hi RocketMQ Community,
> > >
> > > This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
> > >
> > > In order to better implement a lightweight client, @ayanamist proposes
> a
> > > new consumption model, and at the same time transfers the load
> balancing
> > > logic of the original client to the broker, which not only solves the
> > > original queue occupancy problem but also It can also avoid the
> > consumption
> > > delay caused by a certain consumer hangs.
> > >
> > > The vote will be open for at least 72 hours or until a necessary number
> > of
> > > votes are reached.
> > >
> > > Please vote accordingly:
> > >
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove with the reason
> > >
> > >
> > >
> > >
> > > Best Regards!
> > > Henry
> > >
> > > ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
> > >
> > > > # RIP-19 RocketMQ Pop Consuming
> > > >
> > > > # Status
> > > >
> > > > - Current State: Proposed
> > > > - Authors: [ayanamist]([
> > > > https://github.com/ayanamist/](https://github.com/ayanamist/))
> > > > - Shepherds: [duhengforever]([
> > > > https://github.com/duhenglucky/](https://github.com/duhengforever/))
> > > > - Mailing List discussion: dev@rocketmq.apache.org;
> > > > users@rocketmq.apache.org
> > > > - Pull Request: RIP-19
> > > > - Released: -
> > > >
> > > > # Background & Motivation
> > > >
> > > > ### What do we need to do
> > > >
> > > > - Will we add a new module?
> > > >
> > > > No.
> > > >
> > > > - Will we add new APIs?
> > > >
> > > > Yes.
> > > >
> > > > - Will we add new feature?
> > > >
> > > > Yes.
> > > >
> > > > ### Why should we do that
> > > >
> > > > - Are there any problems of our current project?
> > > >
> > > > The current subscription load balancing strategy is based on the
> > > > dimension of message queue. All behaviors are owned by the client
> side.
> > > > There are three main steps:
> > > >
> > > > 1. Each consumer regularly obtains the total number of topic
> > message
> > > > queues and all consumers.
> > > > 2. Using a general algorithm to sort the queues by consumer ip
> and
> > > > queue index to calculate which message queue is allocated to which
> > > > consumer.
> > > > 3. Each consumer pulls messages using allocated orders described
> > > above.
> > > >
> > > > According to this allocation method, if an abnormality occurs in
> a
> > > > consumer (the application itself is abnormal, or a broker is
> upgrading)
> > > so
> > > > that it causes slow subscription, messages will be accumulated, but
> > this
> > > > queue will not be re-allocated to another consumer, so the
> accumulation
> > > > will become more and more serious.
> > > >
> > > >
> > > > Chinese version:
> > > >
> > > > 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
> > > >
> > > > 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
> > > > 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
> > > > 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
> > > >
> > > >
> > > >
> > > >
> > >
> >
> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
> > > >
> > > > - What can we benefit proposed changes?
> > > >
> > > > The accumulated messages will be subscribed by other consumers if
> > one
> > > > consumer behaves abnormally.
> > > >
> > > > Chinese version:
> > > >
> > > > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
> > > >
> > > > # Goals
> > > >
> > > > - What problem is this proposal designed to solve?
> > > >
> > > > The accumulated messages will be subscribed by other consumers if
> > one
> > > > consumer behaves abnormally.
> > > >
> > > > Chinese version:
> > > >
> > > > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
> > > >
> > > > - To what degree should we solve the problem?
> > > >
> > > > This RIP must guarantee below point:
> > > >
> > > > 1. High availablity: Subscription of one message queue will not
> be
> > > > affected by single consumer failure.
> > > > 2. High performance: This implementation affects latency and
> > > throughput
> > > > less than 10%.
> > > >
> > > >
> > > > Chinese version:
> > > >
> > > > 新方案需要保证两点：
> > > >
> > > > 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
> > > > 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
> > > >
> > > > # Non-Goals
> > > >
> > > > - What problem is this proposal NOT designed to solve?
> > > >
> > > > Improve client-side load balancing.
> > > >
> > > > - Are there any limits of this proposal?
> > > >
> > > > Nothing specific.
> > > >
> > > > # Changes
> > > >
> > > > ## Architecture
> > > >
> > > > Current "Pull mode":
> > > > 
> > > >
> > > > Proposed "Pop mode":
> > > > 
> > > >
> > > > Move inter-queue balance of one topic from client side to server
> side.
> > > > Clients make pull request without specified queues to broker, and
> > broker
> > > > fetch messages from queues internally and returns, which ensures one
> > > queue
> > > > will be consumed by multiple clients. The whole behavior is like a
> > queue
> > > > pop process.
> > > >
> > > > It will add a new request command querying queue assignments in
> broker,
> > > and
> > > > add pop-feature-support flag to pull request which makes broker use
> pop
> > > > mode.
> > > >
> > > > ## Interface Design/Change
> > > >
> > > > - Method signature changes
> > > >
> > > > Nothing specific.
> > > >
> > > > - Method behavior changes
> > > >
> > > > Nothing specific.
> > > >
> > > > - CLI command changes
> > > >
> > > > Add `setConsumeMode` for admin to switch between old pull mode
> and
> > > new
> > > > pop mode for one subscription.
> > > >
> > > > - Log format or content changes
> > > >
> > > > Nothing specific.
> > > >
> > > > ## Compatibility, Deprecation, and Migration Plan
> > > >
> > > > - Are backward and forward compatibility taken into consideration?
> > > >
> > > > New RequestCode between client and broker are added, so there
> are 2
> > > > compatibility situations:
> > > >
> > > > 1. old client+new broker: old clients won't make request with
> > > > pop-feature-support flag, so broker will not enable pop mode, which
> > keep
> > > > all things as before.
> > > > 2. new client+old broker: new clients will detect whether broker
> > > > support the new request command querying queue assignments, if not,
> it
> > > will
> > > > fallback to use old pull mode.
> > > >
> > > > - Are there deprecated APIs?
> > > >
> > > > Nothing specific.
> > > >
> > > > - How do we do migration?
> > > >
> > > > Nothing specific.
> > > >
> > > > ## Implementation Outline
> > > >
> > > > We will implement the proposed changes by **2** phases.
> > > >
> > > > ## Phase 1
> > > >
> > > > 1. Implement server-side balance capability in broker
> > > > 2. Implement client-side request using new pop-mode
> > > >
> > > > ## Phase 2
> > > >
> > > > 1. Implement new sdk compatibility with old broker.
> > > > 2. Implement feature detection in broker and client.
> > > >
> > > > # Rejected Alternatives
> > > >
> > > > ## How does alternatives solve the issue you proposed?
> > > >
> > > > Improve client rebalance logic? I don't get a quite good idea.
> > > >
> > > > ## Pros and Cons of alternatives
> > > >
> > > > Client rebalance logic will become quite complicated.
> > > >
> > > > ## Why should we reject above alternatives
> > > >
> > >
> >
>
--
Nothing is impossible
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by ayanamist <ay...@gmail.com>.
I will supplement a work flow later.
Internally, it adds "POP"+"ACK" requests to broker+client interaction which
"POP" fetches messages and "ACK" acknowledges these messages' offsets, then
all flow can be moved from client to broker side. There will be some
changes in push consumer so that it keeps client usage as same as before,
and broker controls whether this consumer group use POP or PULL
On Wed, Jan 20, 2021 at 3:32 PM tiger lee <ti...@apache.org> wrote:
> [ ] +1 approve
> it would make Consumer Client more simple to use. and do u have a work
> flow to show how POP mode work ?
>
> heng du <du...@apache.org> 于2021年1月18日周一 下午2:37写道：
>
> > Hi RocketMQ Community,
> >
> > This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
> >
> > In order to better implement a lightweight client, @ayanamist proposes a
> > new consumption model, and at the same time transfers the load balancing
> > logic of the original client to the broker, which not only solves the
> > original queue occupancy problem but also It can also avoid the
> consumption
> > delay caused by a certain consumer hangs.
> >
> > The vote will be open for at least 72 hours or until a necessary number
> of
> > votes are reached.
> >
> > Please vote accordingly:
> >
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove with the reason
> >
> >
> >
> >
> > Best Regards!
> > Henry
> >
> > ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
> >
> > > # RIP-19 RocketMQ Pop Consuming
> > >
> > > # Status
> > >
> > > - Current State: Proposed
> > > - Authors: [ayanamist]([
> > > https://github.com/ayanamist/](https://github.com/ayanamist/))
> > > - Shepherds: [duhengforever]([
> > > https://github.com/duhenglucky/](https://github.com/duhengforever/))
> > > - Mailing List discussion: dev@rocketmq.apache.org;
> > > users@rocketmq.apache.org
> > > - Pull Request: RIP-19
> > > - Released: -
> > >
> > > # Background & Motivation
> > >
> > > ### What do we need to do
> > >
> > > - Will we add a new module?
> > >
> > > No.
> > >
> > > - Will we add new APIs?
> > >
> > > Yes.
> > >
> > > - Will we add new feature?
> > >
> > > Yes.
> > >
> > > ### Why should we do that
> > >
> > > - Are there any problems of our current project?
> > >
> > > The current subscription load balancing strategy is based on the
> > > dimension of message queue. All behaviors are owned by the client side.
> > > There are three main steps:
> > >
> > > 1. Each consumer regularly obtains the total number of topic
> message
> > > queues and all consumers.
> > > 2. Using a general algorithm to sort the queues by consumer ip and
> > > queue index to calculate which message queue is allocated to which
> > > consumer.
> > > 3. Each consumer pulls messages using allocated orders described
> > above.
> > >
> > > According to this allocation method, if an abnormality occurs in a
> > > consumer (the application itself is abnormal, or a broker is upgrading)
> > so
> > > that it causes slow subscription, messages will be accumulated, but
> this
> > > queue will not be re-allocated to another consumer, so the accumulation
> > > will become more and more serious.
> > >
> > >
> > > Chinese version:
> > >
> > > 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
> > >
> > > 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
> > > 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
> > > 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
> > >
> > >
> > >
> > >
> >
> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
> > >
> > > - What can we benefit proposed changes?
> > >
> > > The accumulated messages will be subscribed by other consumers if
> one
> > > consumer behaves abnormally.
> > >
> > > Chinese version:
> > >
> > > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
> > >
> > > # Goals
> > >
> > > - What problem is this proposal designed to solve?
> > >
> > > The accumulated messages will be subscribed by other consumers if
> one
> > > consumer behaves abnormally.
> > >
> > > Chinese version:
> > >
> > > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
> > >
> > > - To what degree should we solve the problem?
> > >
> > > This RIP must guarantee below point:
> > >
> > > 1. High availablity: Subscription of one message queue will not be
> > > affected by single consumer failure.
> > > 2. High performance: This implementation affects latency and
> > throughput
> > > less than 10%.
> > >
> > >
> > > Chinese version:
> > >
> > > 新方案需要保证两点：
> > >
> > > 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
> > > 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
> > >
> > > # Non-Goals
> > >
> > > - What problem is this proposal NOT designed to solve?
> > >
> > > Improve client-side load balancing.
> > >
> > > - Are there any limits of this proposal?
> > >
> > > Nothing specific.
> > >
> > > # Changes
> > >
> > > ## Architecture
> > >
> > > Current "Pull mode":
> > > 
> > >
> > > Proposed "Pop mode":
> > > 
> > >
> > > Move inter-queue balance of one topic from client side to server side.
> > > Clients make pull request without specified queues to broker, and
> broker
> > > fetch messages from queues internally and returns, which ensures one
> > queue
> > > will be consumed by multiple clients. The whole behavior is like a
> queue
> > > pop process.
> > >
> > > It will add a new request command querying queue assignments in broker,
> > and
> > > add pop-feature-support flag to pull request which makes broker use pop
> > > mode.
> > >
> > > ## Interface Design/Change
> > >
> > > - Method signature changes
> > >
> > > Nothing specific.
> > >
> > > - Method behavior changes
> > >
> > > Nothing specific.
> > >
> > > - CLI command changes
> > >
> > > Add `setConsumeMode` for admin to switch between old pull mode and
> > new
> > > pop mode for one subscription.
> > >
> > > - Log format or content changes
> > >
> > > Nothing specific.
> > >
> > > ## Compatibility, Deprecation, and Migration Plan
> > >
> > > - Are backward and forward compatibility taken into consideration?
> > >
> > > New RequestCode between client and broker are added, so there are 2
> > > compatibility situations:
> > >
> > > 1. old client+new broker: old clients won't make request with
> > > pop-feature-support flag, so broker will not enable pop mode, which
> keep
> > > all things as before.
> > > 2. new client+old broker: new clients will detect whether broker
> > > support the new request command querying queue assignments, if not, it
> > will
> > > fallback to use old pull mode.
> > >
> > > - Are there deprecated APIs?
> > >
> > > Nothing specific.
> > >
> > > - How do we do migration?
> > >
> > > Nothing specific.
> > >
> > > ## Implementation Outline
> > >
> > > We will implement the proposed changes by **2** phases.
> > >
> > > ## Phase 1
> > >
> > > 1. Implement server-side balance capability in broker
> > > 2. Implement client-side request using new pop-mode
> > >
> > > ## Phase 2
> > >
> > > 1. Implement new sdk compatibility with old broker.
> > > 2. Implement feature detection in broker and client.
> > >
> > > # Rejected Alternatives
> > >
> > > ## How does alternatives solve the issue you proposed?
> > >
> > > Improve client rebalance logic? I don't get a quite good idea.
> > >
> > > ## Pros and Cons of alternatives
> > >
> > > Client rebalance logic will become quite complicated.
> > >
> > > ## Why should we reject above alternatives
> > >
> >
>
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by tiger lee <ti...@apache.org>.
[ ] +1 approve
it would make Consumer Client more simple to use. and do u have a work
flow to show how POP mode work ?
heng du <du...@apache.org> 于2021年1月18日周一 下午2:37写道：
> Hi RocketMQ Community,
>
> This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
>
> In order to better implement a lightweight client, @ayanamist proposes a
> new consumption model, and at the same time transfers the load balancing
> logic of the original client to the broker, which not only solves the
> original queue occupancy problem but also It can also avoid the consumption
> delay caused by a certain consumer hangs.
>
> The vote will be open for at least 72 hours or until a necessary number of
> votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
>
>
> Best Regards!
> Henry
>
> ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
>
> > # RIP-19 RocketMQ Pop Consuming
> >
> > # Status
> >
> > - Current State: Proposed
> > - Authors: [ayanamist]([
> > https://github.com/ayanamist/](https://github.com/ayanamist/))
> > - Shepherds: [duhengforever]([
> > https://github.com/duhenglucky/](https://github.com/duhengforever/))
> > - Mailing List discussion: dev@rocketmq.apache.org;
> > users@rocketmq.apache.org
> > - Pull Request: RIP-19
> > - Released: -
> >
> > # Background & Motivation
> >
> > ### What do we need to do
> >
> > - Will we add a new module?
> >
> > No.
> >
> > - Will we add new APIs?
> >
> > Yes.
> >
> > - Will we add new feature?
> >
> > Yes.
> >
> > ### Why should we do that
> >
> > - Are there any problems of our current project?
> >
> > The current subscription load balancing strategy is based on the
> > dimension of message queue. All behaviors are owned by the client side.
> > There are three main steps:
> >
> > 1. Each consumer regularly obtains the total number of topic message
> > queues and all consumers.
> > 2. Using a general algorithm to sort the queues by consumer ip and
> > queue index to calculate which message queue is allocated to which
> > consumer.
> > 3. Each consumer pulls messages using allocated orders described
> above.
> >
> > According to this allocation method, if an abnormality occurs in a
> > consumer (the application itself is abnormal, or a broker is upgrading)
> so
> > that it causes slow subscription, messages will be accumulated, but this
> > queue will not be re-allocated to another consumer, so the accumulation
> > will become more and more serious.
> >
> >
> > Chinese version:
> >
> > 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
> >
> > 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
> > 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
> > 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
> >
> >
> >
> >
> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
> >
> > - What can we benefit proposed changes?
> >
> > The accumulated messages will be subscribed by other consumers if one
> > consumer behaves abnormally.
> >
> > Chinese version:
> >
> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
> >
> > # Goals
> >
> > - What problem is this proposal designed to solve?
> >
> > The accumulated messages will be subscribed by other consumers if one
> > consumer behaves abnormally.
> >
> > Chinese version:
> >
> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
> >
> > - To what degree should we solve the problem?
> >
> > This RIP must guarantee below point:
> >
> > 1. High availablity: Subscription of one message queue will not be
> > affected by single consumer failure.
> > 2. High performance: This implementation affects latency and
> throughput
> > less than 10%.
> >
> >
> > Chinese version:
> >
> > 新方案需要保证两点：
> >
> > 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
> > 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
> >
> > # Non-Goals
> >
> > - What problem is this proposal NOT designed to solve?
> >
> > Improve client-side load balancing.
> >
> > - Are there any limits of this proposal?
> >
> > Nothing specific.
> >
> > # Changes
> >
> > ## Architecture
> >
> > Current "Pull mode":
> > 
> >
> > Proposed "Pop mode":
> > 
> >
> > Move inter-queue balance of one topic from client side to server side.
> > Clients make pull request without specified queues to broker, and broker
> > fetch messages from queues internally and returns, which ensures one
> queue
> > will be consumed by multiple clients. The whole behavior is like a queue
> > pop process.
> >
> > It will add a new request command querying queue assignments in broker,
> and
> > add pop-feature-support flag to pull request which makes broker use pop
> > mode.
> >
> > ## Interface Design/Change
> >
> > - Method signature changes
> >
> > Nothing specific.
> >
> > - Method behavior changes
> >
> > Nothing specific.
> >
> > - CLI command changes
> >
> > Add `setConsumeMode` for admin to switch between old pull mode and
> new
> > pop mode for one subscription.
> >
> > - Log format or content changes
> >
> > Nothing specific.
> >
> > ## Compatibility, Deprecation, and Migration Plan
> >
> > - Are backward and forward compatibility taken into consideration?
> >
> > New RequestCode between client and broker are added, so there are 2
> > compatibility situations:
> >
> > 1. old client+new broker: old clients won't make request with
> > pop-feature-support flag, so broker will not enable pop mode, which keep
> > all things as before.
> > 2. new client+old broker: new clients will detect whether broker
> > support the new request command querying queue assignments, if not, it
> will
> > fallback to use old pull mode.
> >
> > - Are there deprecated APIs?
> >
> > Nothing specific.
> >
> > - How do we do migration?
> >
> > Nothing specific.
> >
> > ## Implementation Outline
> >
> > We will implement the proposed changes by **2** phases.
> >
> > ## Phase 1
> >
> > 1. Implement server-side balance capability in broker
> > 2. Implement client-side request using new pop-mode
> >
> > ## Phase 2
> >
> > 1. Implement new sdk compatibility with old broker.
> > 2. Implement feature detection in broker and client.
> >
> > # Rejected Alternatives
> >
> > ## How does alternatives solve the issue you proposed?
> >
> > Improve client rebalance logic? I don't get a quite good idea.
> >
> > ## Pros and Cons of alternatives
> >
> > Client rebalance logic will become quite complicated.
> >
> > ## Why should we reject above alternatives
> >
>
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by Rongtong Jin <ji...@mails.ucas.ac.cn>.
+1, IMO, it is a good solution to solve some important issues of queue load balancing.
2021-01-18 14:37:27"heng du" <du...@apache.org>写道：
Hi RocketMQ Community,
This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
In order to better implement a lightweight client, @ayanamist proposes a new consumption model, and at the same time transfers the load balancing logic of the original client to the broker, which not only solves the original queue occupancy problem but also It can also avoid the consumption delay caused by a certain consumer hangs.
The vote will be open for at least 72 hours or until a necessary number of votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Best Regards!
Henry
ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
# RIP-19 RocketMQ Pop Consuming
# Status
- Current State: Proposed
- Authors: [ayanamist]([
https://github.com/ayanamist/](https://github.com/ayanamist/))
- Shepherds: [duhengforever]([
https://github.com/duhenglucky/](https://github.com/duhengforever/))
- Mailing List discussion: dev@rocketmq.apache.org;
users@rocketmq.apache.org
- Pull Request: RIP-19
- Released: -
# Background & Motivation
### What do we need to do
- Will we add a new module?
No.
- Will we add new APIs?
Yes.
- Will we add new feature?
Yes.
### Why should we do that
- Are there any problems of our current project?
The current subscription load balancing strategy is based on the
dimension of message queue. All behaviors are owned by the client side.
There are three main steps:
1. Each consumer regularly obtains the total number of topic message
queues and all consumers.
2. Using a general algorithm to sort the queues by consumer ip and
queue index to calculate which message queue is allocated to which consumer.
3. Each consumer pulls messages using allocated orders described above.
According to this allocation method, if an abnormality occurs in a
consumer (the application itself is abnormal, or a broker is upgrading) so
that it causes slow subscription, messages will be accumulated, but this
queue will not be re-allocated to another consumer, so the accumulation
will become more and more serious.
Chinese version:
当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
3. 每个consumer去根据算法分配出来的队列，拉取消息消费
按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
- What can we benefit proposed changes?
The accumulated messages will be subscribed by other consumers if one
consumer behaves abnormally.
Chinese version:
在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
# Goals
- What problem is this proposal designed to solve?
The accumulated messages will be subscribed by other consumers if one
consumer behaves abnormally.
Chinese version:
在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
- To what degree should we solve the problem?
This RIP must guarantee below point:
1. High availablity: Subscription of one message queue will not be
affected by single consumer failure.
2. High performance: This implementation affects latency and throughput
less than 10%.
Chinese version:
新方案需要保证两点：
1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
# Non-Goals
- What problem is this proposal NOT designed to solve?
Improve client-side load balancing.
- Are there any limits of this proposal?
Nothing specific.
# Changes
## Architecture
Current "Pull mode":
Proposed "Pop mode":
Move inter-queue balance of one topic from client side to server side.
Clients make pull request without specified queues to broker, and broker
fetch messages from queues internally and returns, which ensures one queue
will be consumed by multiple clients. The whole behavior is like a queue
pop process.
It will add a new request command querying queue assignments in broker, and
add pop-feature-support flag to pull request which makes broker use pop
mode.
## Interface Design/Change
- Method signature changes
Nothing specific.
- Method behavior changes
Nothing specific.
- CLI command changes
Add `setConsumeMode` for admin to switch between old pull mode and new
pop mode for one subscription.
- Log format or content changes
Nothing specific.
## Compatibility, Deprecation, and Migration Plan
- Are backward and forward compatibility taken into consideration?
New RequestCode between client and broker are added, so there are 2
compatibility situations:
1. old client+new broker: old clients won't make request with
pop-feature-support flag, so broker will not enable pop mode, which keep
all things as before.
2. new client+old broker: new clients will detect whether broker
support the new request command querying queue assignments, if not, it will
fallback to use old pull mode.
- Are there deprecated APIs?
Nothing specific.
- How do we do migration?
Nothing specific.
## Implementation Outline
We will implement the proposed changes by **2** phases.
## Phase 1
1. Implement server-side balance capability in broker
2. Implement client-side request using new pop-mode
## Phase 2
1. Implement new sdk compatibility with old broker.
2. Implement feature detection in broker and client.
# Rejected Alternatives
## How does alternatives solve the issue you proposed?
Improve client rebalance logic? I don't get a quite good idea.
## Pros and Cons of alternatives
Client rebalance logic will become quite complicated.
## Why should we reject above alternatives
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by liqipeng <wl...@163.com>.
+1
> 在 2021年1月18日，下午2:37，heng du <du...@apache.org> 写道：
>
> Hi RocketMQ Community,
>
> This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
>
> In order to better implement a lightweight client, @ayanamist proposes a
> new consumption model, and at the same time transfers the load balancing
> logic of the original client to the broker, which not only solves the
> original queue occupancy problem but also It can also avoid the consumption
> delay caused by a certain consumer hangs.
>
> The vote will be open for at least 72 hours or until a necessary number of
> votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
>
>
> Best Regards!
> Henry
>
> ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
>
>> # RIP-19 RocketMQ Pop Consuming
>>
>> # Status
>>
>> - Current State: Proposed
>> - Authors: [ayanamist]([
>> https://github.com/ayanamist/](https://github.com/ayanamist/))
>> - Shepherds: [duhengforever]([
>> https://github.com/duhenglucky/](https://github.com/duhengforever/))
>> - Mailing List discussion: dev@rocketmq.apache.org;
>> users@rocketmq.apache.org
>> - Pull Request: RIP-19
>> - Released: -
>>
>> # Background & Motivation
>>
>> ### What do we need to do
>>
>> - Will we add a new module?
>>
>> No.
>>
>> - Will we add new APIs?
>>
>> Yes.
>>
>> - Will we add new feature?
>>
>> Yes.
>>
>> ### Why should we do that
>>
>> - Are there any problems of our current project?
>>
>> The current subscription load balancing strategy is based on the
>> dimension of message queue. All behaviors are owned by the client side.
>> There are three main steps:
>>
>> 1. Each consumer regularly obtains the total number of topic message
>> queues and all consumers.
>> 2. Using a general algorithm to sort the queues by consumer ip and
>> queue index to calculate which message queue is allocated to which
>> consumer.
>> 3. Each consumer pulls messages using allocated orders described above.
>>
>> According to this allocation method, if an abnormality occurs in a
>> consumer (the application itself is abnormal, or a broker is upgrading) so
>> that it causes slow subscription, messages will be accumulated, but this
>> queue will not be re-allocated to another consumer, so the accumulation
>> will become more and more serious.
>>
>>
>> Chinese version:
>>
>> 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
>>
>> 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
>> 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
>> 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
>>
>>
>>
>> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
>>
>> - What can we benefit proposed changes?
>>
>> The accumulated messages will be subscribed by other consumers if one
>> consumer behaves abnormally.
>>
>> Chinese version:
>>
>> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>>
>> # Goals
>>
>> - What problem is this proposal designed to solve?
>>
>> The accumulated messages will be subscribed by other consumers if one
>> consumer behaves abnormally.
>>
>> Chinese version:
>>
>> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>>
>> - To what degree should we solve the problem?
>>
>> This RIP must guarantee below point:
>>
>> 1. High availablity: Subscription of one message queue will not be
>> affected by single consumer failure.
>> 2. High performance: This implementation affects latency and throughput
>> less than 10%.
>>
>>
>> Chinese version:
>>
>> 新方案需要保证两点：
>>
>> 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
>> 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
>>
>> # Non-Goals
>>
>> - What problem is this proposal NOT designed to solve?
>>
>> Improve client-side load balancing.
>>
>> - Are there any limits of this proposal?
>>
>> Nothing specific.
>>
>> # Changes
>>
>> ## Architecture
>>
>> Current "Pull mode":
>> 
>>
>> Proposed "Pop mode":
>> 
>>
>> Move inter-queue balance of one topic from client side to server side.
>> Clients make pull request without specified queues to broker, and broker
>> fetch messages from queues internally and returns, which ensures one queue
>> will be consumed by multiple clients. The whole behavior is like a queue
>> pop process.
>>
>> It will add a new request command querying queue assignments in broker, and
>> add pop-feature-support flag to pull request which makes broker use pop
>> mode.
>>
>> ## Interface Design/Change
>>
>> - Method signature changes
>>
>> Nothing specific.
>>
>> - Method behavior changes
>>
>> Nothing specific.
>>
>> - CLI command changes
>>
>> Add `setConsumeMode` for admin to switch between old pull mode and new
>> pop mode for one subscription.
>>
>> - Log format or content changes
>>
>> Nothing specific.
>>
>> ## Compatibility, Deprecation, and Migration Plan
>>
>> - Are backward and forward compatibility taken into consideration?
>>
>> New RequestCode between client and broker are added, so there are 2
>> compatibility situations:
>>
>> 1. old client+new broker: old clients won't make request with
>> pop-feature-support flag, so broker will not enable pop mode, which keep
>> all things as before.
>> 2. new client+old broker: new clients will detect whether broker
>> support the new request command querying queue assignments, if not, it will
>> fallback to use old pull mode.
>>
>> - Are there deprecated APIs?
>>
>> Nothing specific.
>>
>> - How do we do migration?
>>
>> Nothing specific.
>>
>> ## Implementation Outline
>>
>> We will implement the proposed changes by **2** phases.
>>
>> ## Phase 1
>>
>> 1. Implement server-side balance capability in broker
>> 2. Implement client-side request using new pop-mode
>>
>> ## Phase 2
>>
>> 1. Implement new sdk compatibility with old broker.
>> 2. Implement feature detection in broker and client.
>>
>> # Rejected Alternatives
>>
>> ## How does alternatives solve the issue you proposed?
>>
>> Improve client rebalance logic? I don't get a quite good idea.
>>
>> ## Pros and Cons of alternatives
>>
>> Client rebalance logic will become quite complicated.
>>
>> ## Why should we reject above alternatives
>>
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by tiger lee <ti...@apache.org>.
[ ] +1 approve
it would make Consumer Client more simple to use. and do u have a work
flow to show how POP mode work ?
heng du <du...@apache.org> 于2021年1月18日周一 下午2:37写道：
> Hi RocketMQ Community,
>
> This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
>
> In order to better implement a lightweight client, @ayanamist proposes a
> new consumption model, and at the same time transfers the load balancing
> logic of the original client to the broker, which not only solves the
> original queue occupancy problem but also It can also avoid the consumption
> delay caused by a certain consumer hangs.
>
> The vote will be open for at least 72 hours or until a necessary number of
> votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
>
>
> Best Regards!
> Henry
>
> ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
>
> > # RIP-19 RocketMQ Pop Consuming
> >
> > # Status
> >
> > - Current State: Proposed
> > - Authors: [ayanamist]([
> > https://github.com/ayanamist/](https://github.com/ayanamist/))
> > - Shepherds: [duhengforever]([
> > https://github.com/duhenglucky/](https://github.com/duhengforever/))
> > - Mailing List discussion: dev@rocketmq.apache.org;
> > users@rocketmq.apache.org
> > - Pull Request: RIP-19
> > - Released: -
> >
> > # Background & Motivation
> >
> > ### What do we need to do
> >
> > - Will we add a new module?
> >
> > No.
> >
> > - Will we add new APIs?
> >
> > Yes.
> >
> > - Will we add new feature?
> >
> > Yes.
> >
> > ### Why should we do that
> >
> > - Are there any problems of our current project?
> >
> > The current subscription load balancing strategy is based on the
> > dimension of message queue. All behaviors are owned by the client side.
> > There are three main steps:
> >
> > 1. Each consumer regularly obtains the total number of topic message
> > queues and all consumers.
> > 2. Using a general algorithm to sort the queues by consumer ip and
> > queue index to calculate which message queue is allocated to which
> > consumer.
> > 3. Each consumer pulls messages using allocated orders described
> above.
> >
> > According to this allocation method, if an abnormality occurs in a
> > consumer (the application itself is abnormal, or a broker is upgrading)
> so
> > that it causes slow subscription, messages will be accumulated, but this
> > queue will not be re-allocated to another consumer, so the accumulation
> > will become more and more serious.
> >
> >
> > Chinese version:
> >
> > 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
> >
> > 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
> > 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
> > 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
> >
> >
> >
> >
> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
> >
> > - What can we benefit proposed changes?
> >
> > The accumulated messages will be subscribed by other consumers if one
> > consumer behaves abnormally.
> >
> > Chinese version:
> >
> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
> >
> > # Goals
> >
> > - What problem is this proposal designed to solve?
> >
> > The accumulated messages will be subscribed by other consumers if one
> > consumer behaves abnormally.
> >
> > Chinese version:
> >
> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
> >
> > - To what degree should we solve the problem?
> >
> > This RIP must guarantee below point:
> >
> > 1. High availablity: Subscription of one message queue will not be
> > affected by single consumer failure.
> > 2. High performance: This implementation affects latency and
> throughput
> > less than 10%.
> >
> >
> > Chinese version:
> >
> > 新方案需要保证两点：
> >
> > 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
> > 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
> >
> > # Non-Goals
> >
> > - What problem is this proposal NOT designed to solve?
> >
> > Improve client-side load balancing.
> >
> > - Are there any limits of this proposal?
> >
> > Nothing specific.
> >
> > # Changes
> >
> > ## Architecture
> >
> > Current "Pull mode":
> > 
> >
> > Proposed "Pop mode":
> > 
> >
> > Move inter-queue balance of one topic from client side to server side.
> > Clients make pull request without specified queues to broker, and broker
> > fetch messages from queues internally and returns, which ensures one
> queue
> > will be consumed by multiple clients. The whole behavior is like a queue
> > pop process.
> >
> > It will add a new request command querying queue assignments in broker,
> and
> > add pop-feature-support flag to pull request which makes broker use pop
> > mode.
> >
> > ## Interface Design/Change
> >
> > - Method signature changes
> >
> > Nothing specific.
> >
> > - Method behavior changes
> >
> > Nothing specific.
> >
> > - CLI command changes
> >
> > Add `setConsumeMode` for admin to switch between old pull mode and
> new
> > pop mode for one subscription.
> >
> > - Log format or content changes
> >
> > Nothing specific.
> >
> > ## Compatibility, Deprecation, and Migration Plan
> >
> > - Are backward and forward compatibility taken into consideration?
> >
> > New RequestCode between client and broker are added, so there are 2
> > compatibility situations:
> >
> > 1. old client+new broker: old clients won't make request with
> > pop-feature-support flag, so broker will not enable pop mode, which keep
> > all things as before.
> > 2. new client+old broker: new clients will detect whether broker
> > support the new request command querying queue assignments, if not, it
> will
> > fallback to use old pull mode.
> >
> > - Are there deprecated APIs?
> >
> > Nothing specific.
> >
> > - How do we do migration?
> >
> > Nothing specific.
> >
> > ## Implementation Outline
> >
> > We will implement the proposed changes by **2** phases.
> >
> > ## Phase 1
> >
> > 1. Implement server-side balance capability in broker
> > 2. Implement client-side request using new pop-mode
> >
> > ## Phase 2
> >
> > 1. Implement new sdk compatibility with old broker.
> > 2. Implement feature detection in broker and client.
> >
> > # Rejected Alternatives
> >
> > ## How does alternatives solve the issue you proposed?
> >
> > Improve client rebalance logic? I don't get a quite good idea.
> >
> > ## Pros and Cons of alternatives
> >
> > Client rebalance logic will become quite complicated.
> >
> > ## Why should we reject above alternatives
> >
>
Re: [VOTE] RIP-19 RocketMQ Pop Consuming
Posted by "zhuweilin@iie.ac.cn" <zh...@iie.ac.cn>.
+1
zhuweilin@iie.ac.cn
From: heng du
Date: 2021-01-18 14:37
To: dev; ayanamist; users
Subject: [VOTE] RIP-19 RocketMQ Pop Consuming
Hi RocketMQ Community,
This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming.
In order to better implement a lightweight client, @ayanamist proposes a
new consumption model, and at the same time transfers the load balancing
logic of the original client to the broker, which not only solves the
original queue occupancy problem but also It can also avoid the consumption
delay caused by a certain consumer hangs.
The vote will be open for at least 72 hours or until a necessary number of
votes are reached.
Please vote accordingly:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
Best Regards!
Henry
ayanamist <ay...@gmail.com> 于2021年1月8日周五 上午11:25写道：
> # RIP-19 RocketMQ Pop Consuming
>
> # Status
>
> - Current State: Proposed
> - Authors: [ayanamist]([
> https://github.com/ayanamist/](https://github.com/ayanamist/))
> - Shepherds: [duhengforever]([
> https://github.com/duhenglucky/](https://github.com/duhengforever/))
> - Mailing List discussion: dev@rocketmq.apache.org;
> users@rocketmq.apache.org
> - Pull Request: RIP-19
> - Released: -
>
> # Background & Motivation
>
> ### What do we need to do
>
> - Will we add a new module?
>
> No.
>
> - Will we add new APIs?
>
> Yes.
>
> - Will we add new feature?
>
> Yes.
>
> ### Why should we do that
>
> - Are there any problems of our current project?
>
> The current subscription load balancing strategy is based on the
> dimension of message queue. All behaviors are owned by the client side.
> There are three main steps:
>
> 1. Each consumer regularly obtains the total number of topic message
> queues and all consumers.
> 2. Using a general algorithm to sort the queues by consumer ip and
> queue index to calculate which message queue is allocated to which
> consumer.
> 3. Each consumer pulls messages using allocated orders described above.
>
> According to this allocation method, if an abnormality occurs in a
> consumer (the application itself is abnormal, or a broker is upgrading) so
> that it causes slow subscription, messages will be accumulated, but this
> queue will not be re-allocated to another consumer, so the accumulation
> will become more and more serious.
>
>
> Chinese version:
>
> 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步：
>
> 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数
> 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列
> 3. 每个consumer去根据算法分配出来的队列，拉取消息消费
>
>
>
> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。
>
> - What can we benefit proposed changes?
>
> The accumulated messages will be subscribed by other consumers if one
> consumer behaves abnormally.
>
> Chinese version:
>
> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>
> # Goals
>
> - What problem is this proposal designed to solve?
>
> The accumulated messages will be subscribed by other consumers if one
> consumer behaves abnormally.
>
> Chinese version:
>
> 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。
>
> - To what degree should we solve the problem?
>
> This RIP must guarantee below point:
>
> 1. High availablity: Subscription of one message queue will not be
> affected by single consumer failure.
> 2. High performance: This implementation affects latency and throughput
> less than 10%.
>
>
> Chinese version:
>
> 新方案需要保证两点：
>
> 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响
> 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内
>
> # Non-Goals
>
> - What problem is this proposal NOT designed to solve?
>
> Improve client-side load balancing.
>
> - Are there any limits of this proposal?
>
> Nothing specific.
>
> # Changes
>
> ## Architecture
>
> Current "Pull mode":
> 
>
> Proposed "Pop mode":
> 
>
> Move inter-queue balance of one topic from client side to server side.
> Clients make pull request without specified queues to broker, and broker
> fetch messages from queues internally and returns, which ensures one queue
> will be consumed by multiple clients. The whole behavior is like a queue
> pop process.
>
> It will add a new request command querying queue assignments in broker, and
> add pop-feature-support flag to pull request which makes broker use pop
> mode.
>
> ## Interface Design/Change
>
> - Method signature changes
>
> Nothing specific.
>
> - Method behavior changes
>
> Nothing specific.
>
> - CLI command changes
>
> Add `setConsumeMode` for admin to switch between old pull mode and new
> pop mode for one subscription.
>
> - Log format or content changes
>
> Nothing specific.
>
> ## Compatibility, Deprecation, and Migration Plan
>
> - Are backward and forward compatibility taken into consideration?
>
> New RequestCode between client and broker are added, so there are 2
> compatibility situations:
>
> 1. old client+new broker: old clients won't make request with
> pop-feature-support flag, so broker will not enable pop mode, which keep
> all things as before.
> 2. new client+old broker: new clients will detect whether broker
> support the new request command querying queue assignments, if not, it will
> fallback to use old pull mode.
>
> - Are there deprecated APIs?
>
> Nothing specific.
>
> - How do we do migration?
>
> Nothing specific.
>
> ## Implementation Outline
>
> We will implement the proposed changes by **2** phases.
>
> ## Phase 1
>
> 1. Implement server-side balance capability in broker
> 2. Implement client-side request using new pop-mode
>
> ## Phase 2
>
> 1. Implement new sdk compatibility with old broker.
> 2. Implement feature detection in broker and client.
>
> # Rejected Alternatives
>
> ## How does alternatives solve the issue you proposed?
>
> Improve client rebalance logic? I don't get a quite good idea.
>
> ## Pros and Cons of alternatives
>
> Client rebalance logic will become quite complicated.
>
> ## Why should we reject above alternatives
>