You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by ayanamist <ay...@gmail.com> on 2021/05/18 02:13:57 UTC

[DISCUSS] RIP-21 RocketMQ Logical Queue

Below is Markdown text with some GFM syntax.

# Status
- Current State: Proposed
- Authors: [ayanamist](https://github.com/ayanamist)
- Shepherds: duhengforever@apache.org
- Mailing List discussion: dev@rocketmq.apache.org
- Pull Request: #PR_NUMBER
- Released: <relased_version>
# Background & Motivation
What do we need to do
- Will we add a new module?
No.
- Will we add new APIs?
It will add a new constant to mock broker names for logical queues.
- Will we add a new feature?
Yes.

Why should we do that
- Are there any problems with our current project?
Currently, the MessageQueue of RocketMQ is coupled with broker name, which
results that the queue number will change if broker number increases or
decreases, which causes all queues to rebalance, which may cause service
disruption like flink job restarts in minutes.
- What can we benefit from proposed changes?
The number of logical queues is not related with the number of brokers: We
can increase broker number without changing logical queue number, moreover,
we can increase logical queue number without deploying a new broker.
# Goals
- What problem is this proposal designed to solve?
Provide an abstraction, logical queue, to decouple between queue number and
broker number.
- To what degree should we solve the problem?
We should not hurt availability or performance in the implementation.
# Non-Goals
- What problem is this proposal NOT designed to solve?
We will not improve the mechanism of queues rebalance.
- Are there any limits of this proposal?
Only newer clients with changes in this proposal will benefit.
# Changes
## Architecture

We use one or more MessageQueue to make one LogicalQueue; One LogicalQueue
is unique in one topic, and one MessageQueue belongs to one and only one
LogicalQueue.

| brokerName | MessageQueue | LogicalQueue |
|------------|--------------|--------------|
| broker1    | 0            | 0            |
| broker1    | 1            | 1            |
| broker2    | 0            | 2            |
| broker2    | 1            | 3            |

After one LogicalQueue migrated from broker1 to broker2, there will be two
MessageQueues for one LogicalQueue. We only migrate mapping but not actual
data, so broker1 is still serving for old data consuming but not data
producing.

| brokerName | MessageQueue | LogicalQueue | QueueStatus |
|------------|--------------|--------------|--------------------|
| **broker1**    | **0**            | **0(0-100)**         | **ReadOnly**
        |
| broker1    | 1            | 1            | Normal             |
| broker2    | 0            | 2            | Normal             |
| broker2    | 1            | 3            | Normal             |
| **broker2**    | **2**            | **0(101-)**         | **Normal**
        |

After broker1 cleans all data from the commit log and consume queue,
QueueStatus becomes Expired(neither readable nor writable).

| brokerName | MessageQueue | LogicalQueue | QueueStatus |
|------------|--------------|--------------|-------------|
| **broker1**    | **0**            | **0(-)**         | **Expired**     |
| broker1    | 1            | 1            | Normal      |
| broker2    | 0            | 2            | Normal      |
| broker2    | 1            | 3            | Normal      |
| broker2    | 2            | 0(101-)      | Normal      |

If this LogicalQueue is migrated back to broker1, it will reuse this
expired MessageQueue

| brokerName | MessageQueue | LogicalQueue | QueueStatus |
|------------|--------------|--------------|-------------|
| **broker1**    | **0**            | **0(201-)**      | **Normal**      |
| broker1    | 1            | 1            | Normal      |
| broker2    | 0            | 2            | Normal      |
| broker2    | 1            | 3            | Normal      |
| **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |

If this LogicalQueue is migrated back to broker1 while MessageQueue not
expired, it will create a new MessageQueue

| brokerName | MessageQueue | LogicalQueue | QueueStatus |
|------------|--------------|--------------|-------------|
| **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**    |
| broker1    | 1            | 1            | Normal      |
| **broker1**    | **2**            | **0(201-)**      | **Normal**      |
| broker2    | 0            | 2            | Normal      |
| broker2    | 1            | 3            | Normal      |
| **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |

If broker2 is offlined, all LogicalQueue in this broker should be migrated
away.

| brokerName | MessageQueue | LogicalQueue | QueueStatus |
|------------|--------------|--------------|-------------|
| broker1    | 0            | 0            | Normal      |
| broker1    | 1            | 1            | Normal      |
| **broker1**    | **2**            | **2(101-)**      | **Normal**      |
| **broker1**    | **3**            | **3(101-)**      | **Normal**      |
| **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**    |
| **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**    |

When all data including commit log and consume queue in broker2 are
cleaned, broker2 can be removed.

| brokerName | MessageQueue | LogicalQueue | QueueStatus |
|------------|--------------|--------------|-------------|
| broker1    | 0            | 0            | Normal      |
| broker1    | 1            | 1            | Normal      |
| **broker1**    | **2**            | **2(101-)**     | **Normal**      |
| **broker1**    | **3**            | **3(101-)**      | **Normal**      |

When a new broker is deployed, we can migrate some LogicalQueues to this
broker to spare some producing traffic.

| brokerName | MessageQueue | LogicalQueue | QueueStatus |
|------------|--------------|--------------|-------------|
| broker1    | 0            | 0            | Normal      |
| broker1    | 1            | 1            | Normal      |
| **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**    |
| **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**    |
| **broker3**    | **0**            | **2(201-)**      | **Normal**      |
| **broker3**    | **1**            | **3(201-)**      | **Normal**      |

All mapping data are stored separately in brokers, and via registerBroker
to be gathered in namesrv; Each broker only stores its own logical queue
mapping but not other one's. All management operations should be invoked by
CLI or direct rpc command, automatic operation support is missing now, and
require [RIP-18](
https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade)
to be implemented first.

## Interface Design/Change
- Method signature changes
No.
- Method behavior changes
When a topic is enabled LogicalQueue support, broker name of MessageQueue
result returned by some methods like `fetchSubscribeMessageQueues` or
`fetchPublishMessageQueues` will be a fake one, since LogicalQueue does not
have broker name concept.
- CLI command changes
Add some operation command for LogicalQueue, like
`UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
`QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`.
Also `updateTopic` adds a new parameter to enable LogicalQueue support
immediately after the topic is created.
- Log format or content changes
No.
## Compatibility, Deprecation, and Migration Plan
- Are backward and forward compatibility taken into consideration?
Yes.
- Everything will work well if no topic is enabled LogicalQueue support,
whether on old/new broker/namesrv/client.
- LogicalQueue support will work only under new broker+namesrv+client,
otherwise, everything will work like no LogicalQueue support.
- Are there deprecated APIs?
No.
- How do we do migration?
No need to migrate, this is a feature which needs to be enabled manually.
## Implementation Outline
We will implement the proposed changes by 2 phases.
### Phase 1 basic function
1. Implement LogicalQueue routes query and merge mechanism in namesrv.
2. Implement LogicalQueue mapping report from broker to namesrv.
3. Implement LogicalQueue support in the client.
4. Implement LogicalQueue migration in broker.
5. Implement LogicalQueue admin processor and CLI.
### Phase 2 compatibility
1. Implement proxy request+response for old client in broker.
2. Implement intermediate state(support is partly enabled in some brokers
but not all) protection in namesrv.
# Rejected Alternatives
- How does alternatives solve the issue you proposed?
Implement a whole new Logical Queue architecture from scratch, this
absolutely will solve the problem.
- Pros and Cons of alternatives
Pros: from-scratch way does not bring anything good.
Cons: from-scratch way will break existent concept, add much more
complexity to code and not user-friendly.
- Why should we reject the above alternatives
It does no good.

[RESULT][VOTE] RIP-21 RocketMQ Logical Queue

Posted by heng du <du...@apache.org>.
Hello RocketMQ Community,

This is the vote result for the kickoff of RIP-21 Apache RocketMQ logic
queue support, and it has been passed with [3] binding +1s, [3] non-binding:

*Binding votes +1s:*

heng du (duhengforever)

dongeforever(dongeforever)

Rongtongjin(Jinrongtong)


*Non-binding votes +1s:*

zhiboli
xu16 Zhang
炼龙




This RIP will be accepted and its status will be updated to RocketMQ  Wiki
soon.

Thanks,
The Apache RocketMQ Team

heng du <du...@apache.org> 于2021年5月29日周六 上午9:55写道：

> Hi RocketMQ Community,
>
> This is the vote for the kickoff of RIP-21 RocketMQ logic queue.
>
> In order to better meet the needs of stream computing and scaling, the
> concept of logical queues is proposed in this RIP, which enables RocketMQ
> to have the ability to scale in seconds without changing the number of
> queues.
>
> The vote will be open for at least 72 hours or until a necessary number
> of votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
>
> ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：
>
>> Below is Markdown text with some GFM syntax.
>>
>> # Status
>> - Current State: Proposed
>> - Authors: [ayanamist](https://github.com/ayanamist)
>> - Shepherds: duhengforever@apache.org
>> - Mailing List discussion: dev@rocketmq.apache.org
>> - Pull Request: #PR_NUMBER
>> - Released: <relased_version>
>> # Background & Motivation
>> What do we need to do
>> - Will we add a new module?
>> No.
>> - Will we add new APIs?
>> It will add a new constant to mock broker names for logical queues.
>> - Will we add a new feature?
>> Yes.
>>
>> Why should we do that
>> - Are there any problems with our current project?
>> Currently, the MessageQueue of RocketMQ is coupled with broker name, which
>> results that the queue number will change if broker number increases or
>> decreases, which causes all queues to rebalance, which may cause service
>> disruption like flink job restarts in minutes.
>> - What can we benefit from proposed changes?
>> The number of logical queues is not related with the number of brokers: We
>> can increase broker number without changing logical queue number,
>> moreover,
>> we can increase logical queue number without deploying a new broker.
>> # Goals
>> - What problem is this proposal designed to solve?
>> Provide an abstraction, logical queue, to decouple between queue number
>> and
>> broker number.
>> - To what degree should we solve the problem?
>> We should not hurt availability or performance in the implementation.
>> # Non-Goals
>> - What problem is this proposal NOT designed to solve?
>> We will not improve the mechanism of queues rebalance.
>> - Are there any limits of this proposal?
>> Only newer clients with changes in this proposal will benefit.
>> # Changes
>> ## Architecture
>>
>> We use one or more MessageQueue to make one LogicalQueue; One LogicalQueue
>> is unique in one topic, and one MessageQueue belongs to one and only one
>> LogicalQueue.
>>
>> | brokerName | MessageQueue | LogicalQueue |
>> |------------|--------------|--------------|
>> | broker1    | 0            | 0            |
>> | broker1    | 1            | 1            |
>> | broker2    | 0            | 2            |
>> | broker2    | 1            | 3            |
>>
>> After one LogicalQueue migrated from broker1 to broker2, there will be two
>> MessageQueues for one LogicalQueue. We only migrate mapping but not actual
>> data, so broker1 is still serving for old data consuming but not data
>> producing.
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|--------------------|
>> | **broker1**    | **0**            | **0(0-100)**         | **ReadOnly**
>>         |
>> | broker1    | 1            | 1            | Normal             |
>> | broker2    | 0            | 2            | Normal             |
>> | broker2    | 1            | 3            | Normal             |
>> | **broker2**    | **2**            | **0(101-)**         | **Normal**
>>         |
>>
>> After broker1 cleans all data from the commit log and consume queue,
>> QueueStatus becomes Expired(neither readable nor writable).
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | **broker1**    | **0**            | **0(-)**         | **Expired**     |
>> | broker1    | 1            | 1            | Normal      |
>> | broker2    | 0            | 2            | Normal      |
>> | broker2    | 1            | 3            | Normal      |
>> | broker2    | 2            | 0(101-)      | Normal      |
>>
>> If this LogicalQueue is migrated back to broker1, it will reuse this
>> expired MessageQueue
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | **broker1**    | **0**            | **0(201-)**      | **Normal**      |
>> | broker1    | 1            | 1            | Normal      |
>> | broker2    | 0            | 2            | Normal      |
>> | broker2    | 1            | 3            | Normal      |
>> | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
>>
>> If this LogicalQueue is migrated back to broker1 while MessageQueue not
>> expired, it will create a new MessageQueue
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**    |
>> | broker1    | 1            | 1            | Normal      |
>> | **broker1**    | **2**            | **0(201-)**      | **Normal**      |
>> | broker2    | 0            | 2            | Normal      |
>> | broker2    | 1            | 3            | Normal      |
>> | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
>>
>> If broker2 is offlined, all LogicalQueue in this broker should be migrated
>> away.
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | broker1    | 0            | 0            | Normal      |
>> | broker1    | 1            | 1            | Normal      |
>> | **broker1**    | **2**            | **2(101-)**      | **Normal**      |
>> | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
>> | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**    |
>> | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**    |
>>
>> When all data including commit log and consume queue in broker2 are
>> cleaned, broker2 can be removed.
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | broker1    | 0            | 0            | Normal      |
>> | broker1    | 1            | 1            | Normal      |
>> | **broker1**    | **2**            | **2(101-)**     | **Normal**      |
>> | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
>>
>> When a new broker is deployed, we can migrate some LogicalQueues to this
>> broker to spare some producing traffic.
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | broker1    | 0            | 0            | Normal      |
>> | broker1    | 1            | 1            | Normal      |
>> | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**    |
>> | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**    |
>> | **broker3**    | **0**            | **2(201-)**      | **Normal**      |
>> | **broker3**    | **1**            | **3(201-)**      | **Normal**      |
>>
>> All mapping data are stored separately in brokers, and via registerBroker
>> to be gathered in namesrv; Each broker only stores its own logical queue
>> mapping but not other one's. All management operations should be invoked
>> by
>> CLI or direct rpc command, automatic operation support is missing now, and
>> require [RIP-18](
>>
>> https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
>> )
>> to be implemented first.
>>
>> ## Interface Design/Change
>> - Method signature changes
>> No.
>> - Method behavior changes
>> When a topic is enabled LogicalQueue support, broker name of MessageQueue
>> result returned by some methods like `fetchSubscribeMessageQueues` or
>> `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does
>> not
>> have broker name concept.
>> - CLI command changes
>> Add some operation command for LogicalQueue, like
>> `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
>> `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`.
>> Also `updateTopic` adds a new parameter to enable LogicalQueue support
>> immediately after the topic is created.
>> - Log format or content changes
>> No.
>> ## Compatibility, Deprecation, and Migration Plan
>> - Are backward and forward compatibility taken into consideration?
>> Yes.
>> - Everything will work well if no topic is enabled LogicalQueue support,
>> whether on old/new broker/namesrv/client.
>> - LogicalQueue support will work only under new broker+namesrv+client,
>> otherwise, everything will work like no LogicalQueue support.
>> - Are there deprecated APIs?
>> No.
>> - How do we do migration?
>> No need to migrate, this is a feature which needs to be enabled manually.
>> ## Implementation Outline
>> We will implement the proposed changes by 2 phases.
>> ### Phase 1 basic function
>> 1. Implement LogicalQueue routes query and merge mechanism in namesrv.
>> 2. Implement LogicalQueue mapping report from broker to namesrv.
>> 3. Implement LogicalQueue support in the client.
>> 4. Implement LogicalQueue migration in broker.
>> 5. Implement LogicalQueue admin processor and CLI.
>> ### Phase 2 compatibility
>> 1. Implement proxy request+response for old client in broker.
>> 2. Implement intermediate state(support is partly enabled in some brokers
>> but not all) protection in namesrv.
>> # Rejected Alternatives
>> - How does alternatives solve the issue you proposed?
>> Implement a whole new Logical Queue architecture from scratch, this
>> absolutely will solve the problem.
>> - Pros and Cons of alternatives
>> Pros: from-scratch way does not bring anything good.
>> Cons: from-scratch way will break existent concept, add much more
>> complexity to code and not user-friendly.
>> - Why should we reject the above alternatives
>> It does no good.
>>
>

Re: [External Mail]Re: [VOTE] RIP-21 RocketMQ Logical Queue

Posted by heng du <du...@apache.org>.
+1

dongeforever <do...@apache.org> 于2021年6月1日周二 下午2:18写道：

> +1
> Nice.
> LogicQueue is a great feature and will bring new capabilities for RocketMQ.
>
> Xu16 Zhang 张旭 <zh...@xiaomi.com.invalid> 于2021年6月1日周二 下午12:04写道：
>
> > +1
> >
> >
> >
> > --------
> > 张旭
> > 人工智能与云平台-云技术-计算平台组
> > 电话: 15011056041
> >
> > ________________________________________
> > 发件人: zhibo li <os...@gmail.com>
> > 发送时间: 2021年5月31日 15:00
> > 收件人: dev@rocketmq.apache.org
> > 主题: [External Mail]Re: [VOTE] RIP-21 RocketMQ Logical Queue
> >
> > *外部邮件，谨慎处理 | This message originated from outside of XIAOMI. Please treat
> > this email with caution*
> >
> >
> > +1
> >
> > heng du <du...@apache.org> 于2021年5月29日周六 上午9:55写道：
> >
> > > Hi RocketMQ Community,
> > >
> > > This is the vote for the kickoff of RIP-21 RocketMQ logic queue.
> > >
> > > In order to better meet the needs of stream computing and scaling, the
> > > concept of logical queues is proposed in this RIP, which enables
> RocketMQ
> > > to have the ability to scale in seconds without changing the number of
> > > queues.
> > >
> > > The vote will be open for at least 72 hours or until a necessary number
> > > of votes are reached.
> > >
> > > Please vote accordingly:
> > >
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove with the reason
> > >
> > >
> > >
> > > ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：
> > >
> > > > Below is Markdown text with some GFM syntax.
> > > >
> > > > # Status
> > > > - Current State: Proposed
> > > > - Authors: [ayanamist](https://github.com/ayanamist)
> > > > - Shepherds: duhengforever@apache.org
> > > > - Mailing List discussion: dev@rocketmq.apache.org
> > > > - Pull Request: #PR_NUMBER
> > > > - Released: <relased_version>
> > > > # Background & Motivation
> > > > What do we need to do
> > > > - Will we add a new module?
> > > > No.
> > > > - Will we add new APIs?
> > > > It will add a new constant to mock broker names for logical queues.
> > > > - Will we add a new feature?
> > > > Yes.
> > > >
> > > > Why should we do that
> > > > - Are there any problems with our current project?
> > > > Currently, the MessageQueue of RocketMQ is coupled with broker name,
> > > which
> > > > results that the queue number will change if broker number increases
> or
> > > > decreases, which causes all queues to rebalance, which may cause
> > service
> > > > disruption like flink job restarts in minutes.
> > > > - What can we benefit from proposed changes?
> > > > The number of logical queues is not related with the number of
> brokers:
> > > We
> > > > can increase broker number without changing logical queue number,
> > > moreover,
> > > > we can increase logical queue number without deploying a new broker.
> > > > # Goals
> > > > - What problem is this proposal designed to solve?
> > > > Provide an abstraction, logical queue, to decouple between queue
> number
> > > and
> > > > broker number.
> > > > - To what degree should we solve the problem?
> > > > We should not hurt availability or performance in the implementation.
> > > > # Non-Goals
> > > > - What problem is this proposal NOT designed to solve?
> > > > We will not improve the mechanism of queues rebalance.
> > > > - Are there any limits of this proposal?
> > > > Only newer clients with changes in this proposal will benefit.
> > > > # Changes
> > > > ## Architecture
> > > >
> > > > We use one or more MessageQueue to make one LogicalQueue; One
> > > LogicalQueue
> > > > is unique in one topic, and one MessageQueue belongs to one and only
> > one
> > > > LogicalQueue.
> > > >
> > > > | brokerName | MessageQueue | LogicalQueue |
> > > > |------------|--------------|--------------|
> > > > | broker1    | 0            | 0            |
> > > > | broker1    | 1            | 1            |
> > > > | broker2    | 0            | 2            |
> > > > | broker2    | 1            | 3            |
> > > >
> > > > After one LogicalQueue migrated from broker1 to broker2, there will
> be
> > > two
> > > > MessageQueues for one LogicalQueue. We only migrate mapping but not
> > > actual
> > > > data, so broker1 is still serving for old data consuming but not data
> > > > producing.
> > > >
> > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > > |------------|--------------|--------------|--------------------|
> > > > | **broker1**    | **0**            | **0(0-100)**         |
> > **ReadOnly**
> > > >         |
> > > > | broker1    | 1            | 1            | Normal             |
> > > > | broker2    | 0            | 2            | Normal             |
> > > > | broker2    | 1            | 3            | Normal             |
> > > > | **broker2**    | **2**            | **0(101-)**         |
> **Normal**
> > > >         |
> > > >
> > > > After broker1 cleans all data from the commit log and consume queue,
> > > > QueueStatus becomes Expired(neither readable nor writable).
> > > >
> > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > > |------------|--------------|--------------|-------------|
> > > > | **broker1**    | **0**            | **0(-)**         | **Expired**
> > >  |
> > > > | broker1    | 1            | 1            | Normal      |
> > > > | broker2    | 0            | 2            | Normal      |
> > > > | broker2    | 1            | 3            | Normal      |
> > > > | broker2    | 2            | 0(101-)      | Normal      |
> > > >
> > > > If this LogicalQueue is migrated back to broker1, it will reuse this
> > > > expired MessageQueue
> > > >
> > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > > |------------|--------------|--------------|-------------|
> > > > | **broker1**    | **0**            | **0(201-)**      | **Normal**
> > > |
> > > > | broker1    | 1            | 1            | Normal      |
> > > > | broker2    | 0            | 2            | Normal      |
> > > > | broker2    | 1            | 3            | Normal      |
> > > > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**
> > > |
> > > >
> > > > If this LogicalQueue is migrated back to broker1 while MessageQueue
> not
> > > > expired, it will create a new MessageQueue
> > > >
> > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > > |------------|--------------|--------------|-------------|
> > > > | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**
> > > |
> > > > | broker1    | 1            | 1            | Normal      |
> > > > | **broker1**    | **2**            | **0(201-)**      | **Normal**
> > > |
> > > > | broker2    | 0            | 2            | Normal      |
> > > > | broker2    | 1            | 3            | Normal      |
> > > > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**
> > > |
> > > >
> > > > If broker2 is offlined, all LogicalQueue in this broker should be
> > > migrated
> > > > away.
> > > >
> > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > > |------------|--------------|--------------|-------------|
> > > > | broker1    | 0            | 0            | Normal      |
> > > > | broker1    | 1            | 1            | Normal      |
> > > > | **broker1**    | **2**            | **2(101-)**      | **Normal**
> > > |
> > > > | **broker1**    | **3**            | **3(101-)**      | **Normal**
> > > |
> > > > | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**
> > > |
> > > > | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**
> > > |
> > > >
> > > > When all data including commit log and consume queue in broker2 are
> > > > cleaned, broker2 can be removed.
> > > >
> > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > > |------------|--------------|--------------|-------------|
> > > > | broker1    | 0            | 0            | Normal      |
> > > > | broker1    | 1            | 1            | Normal      |
> > > > | **broker1**    | **2**            | **2(101-)**     | **Normal**
> >   |
> > > > | **broker1**    | **3**            | **3(101-)**      | **Normal**
> > > |
> > > >
> > > > When a new broker is deployed, we can migrate some LogicalQueues to
> > this
> > > > broker to spare some producing traffic.
> > > >
> > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > > |------------|--------------|--------------|-------------|
> > > > | broker1    | 0            | 0            | Normal      |
> > > > | broker1    | 1            | 1            | Normal      |
> > > > | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**
> > > |
> > > > | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**
> > > |
> > > > | **broker3**    | **0**            | **2(201-)**      | **Normal**
> > > |
> > > > | **broker3**    | **1**            | **3(201-)**      | **Normal**
> > > |
> > > >
> > > > All mapping data are stored separately in brokers, and via
> > registerBroker
> > > > to be gathered in namesrv; Each broker only stores its own logical
> > queue
> > > > mapping but not other one's. All management operations should be
> > invoked
> > > by
> > > > CLI or direct rpc command, automatic operation support is missing
> now,
> > > and
> > > > require [RIP-18](
> > > >
> > > >
> > >
> >
> https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
> > > > )
> > > > to be implemented first.
> > > >
> > > > ## Interface Design/Change
> > > > - Method signature changes
> > > > No.
> > > > - Method behavior changes
> > > > When a topic is enabled LogicalQueue support, broker name of
> > MessageQueue
> > > > result returned by some methods like `fetchSubscribeMessageQueues` or
> > > > `fetchPublishMessageQueues` will be a fake one, since LogicalQueue
> does
> > > not
> > > > have broker name concept.
> > > > - CLI command changes
> > > > Add some operation command for LogicalQueue, like
> > > > `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
> > > > `QueryLogicalQueueMapping` `UpdateLogicalQueueNum`
> > `MigrateLogicalQueue`.
> > > > Also `updateTopic` adds a new parameter to enable LogicalQueue
> support
> > > > immediately after the topic is created.
> > > > - Log format or content changes
> > > > No.
> > > > ## Compatibility, Deprecation, and Migration Plan
> > > > - Are backward and forward compatibility taken into consideration?
> > > > Yes.
> > > > - Everything will work well if no topic is enabled LogicalQueue
> > support,
> > > > whether on old/new broker/namesrv/client.
> > > > - LogicalQueue support will work only under new
> broker+namesrv+client,
> > > > otherwise, everything will work like no LogicalQueue support.
> > > > - Are there deprecated APIs?
> > > > No.
> > > > - How do we do migration?
> > > > No need to migrate, this is a feature which needs to be enabled
> > manually.
> > > > ## Implementation Outline
> > > > We will implement the proposed changes by 2 phases.
> > > > ### Phase 1 basic function
> > > > 1. Implement LogicalQueue routes query and merge mechanism in
> namesrv.
> > > > 2. Implement LogicalQueue mapping report from broker to namesrv.
> > > > 3. Implement LogicalQueue support in the client.
> > > > 4. Implement LogicalQueue migration in broker.
> > > > 5. Implement LogicalQueue admin processor and CLI.
> > > > ### Phase 2 compatibility
> > > > 1. Implement proxy request+response for old client in broker.
> > > > 2. Implement intermediate state(support is partly enabled in some
> > brokers
> > > > but not all) protection in namesrv.
> > > > # Rejected Alternatives
> > > > - How does alternatives solve the issue you proposed?
> > > > Implement a whole new Logical Queue architecture from scratch, this
> > > > absolutely will solve the problem.
> > > > - Pros and Cons of alternatives
> > > > Pros: from-scratch way does not bring anything good.
> > > > Cons: from-scratch way will break existent concept, add much more
> > > > complexity to code and not user-friendly.
> > > > - Why should we reject the above alternatives
> > > > It does no good.
> > > >
> > >
> >
> #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
> > This e-mail and its attachments contain confidential information from
> > XIAOMI, which is intended only for the person or entity whose address is
> > listed above. Any use of the information contained herein in any way
> > (including, but not limited to, total or partial disclosure,
> reproduction,
> > or dissemination) by persons other than the intended recipient(s) is
> > prohibited. If you receive this e-mail in error, please notify the sender
> > by phone or email immediately and delete it!******/#
> >
>

Re: [External Mail]Re: [VOTE] RIP-21 RocketMQ Logical Queue

Posted by dongeforever <do...@apache.org>.
+1
Nice.
LogicQueue is a great feature and will bring new capabilities for RocketMQ.

Xu16 Zhang 张旭 <zh...@xiaomi.com.invalid> 于2021年6月1日周二 下午12:04写道：

> +1
>
>
>
> --------
> 张旭
> 人工智能与云平台-云技术-计算平台组
> 电话: 15011056041
>
> ________________________________________
> 发件人: zhibo li <os...@gmail.com>
> 发送时间: 2021年5月31日 15:00
> 收件人: dev@rocketmq.apache.org
> 主题: [External Mail]Re: [VOTE] RIP-21 RocketMQ Logical Queue
>
> *外部邮件，谨慎处理 | This message originated from outside of XIAOMI. Please treat
> this email with caution*
>
>
> +1
>
> heng du <du...@apache.org> 于2021年5月29日周六 上午9:55写道：
>
> > Hi RocketMQ Community,
> >
> > This is the vote for the kickoff of RIP-21 RocketMQ logic queue.
> >
> > In order to better meet the needs of stream computing and scaling, the
> > concept of logical queues is proposed in this RIP, which enables RocketMQ
> > to have the ability to scale in seconds without changing the number of
> > queues.
> >
> > The vote will be open for at least 72 hours or until a necessary number
> > of votes are reached.
> >
> > Please vote accordingly:
> >
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove with the reason
> >
> >
> >
> > ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：
> >
> > > Below is Markdown text with some GFM syntax.
> > >
> > > # Status
> > > - Current State: Proposed
> > > - Authors: [ayanamist](https://github.com/ayanamist)
> > > - Shepherds: duhengforever@apache.org
> > > - Mailing List discussion: dev@rocketmq.apache.org
> > > - Pull Request: #PR_NUMBER
> > > - Released: <relased_version>
> > > # Background & Motivation
> > > What do we need to do
> > > - Will we add a new module?
> > > No.
> > > - Will we add new APIs?
> > > It will add a new constant to mock broker names for logical queues.
> > > - Will we add a new feature?
> > > Yes.
> > >
> > > Why should we do that
> > > - Are there any problems with our current project?
> > > Currently, the MessageQueue of RocketMQ is coupled with broker name,
> > which
> > > results that the queue number will change if broker number increases or
> > > decreases, which causes all queues to rebalance, which may cause
> service
> > > disruption like flink job restarts in minutes.
> > > - What can we benefit from proposed changes?
> > > The number of logical queues is not related with the number of brokers:
> > We
> > > can increase broker number without changing logical queue number,
> > moreover,
> > > we can increase logical queue number without deploying a new broker.
> > > # Goals
> > > - What problem is this proposal designed to solve?
> > > Provide an abstraction, logical queue, to decouple between queue number
> > and
> > > broker number.
> > > - To what degree should we solve the problem?
> > > We should not hurt availability or performance in the implementation.
> > > # Non-Goals
> > > - What problem is this proposal NOT designed to solve?
> > > We will not improve the mechanism of queues rebalance.
> > > - Are there any limits of this proposal?
> > > Only newer clients with changes in this proposal will benefit.
> > > # Changes
> > > ## Architecture
> > >
> > > We use one or more MessageQueue to make one LogicalQueue; One
> > LogicalQueue
> > > is unique in one topic, and one MessageQueue belongs to one and only
> one
> > > LogicalQueue.
> > >
> > > | brokerName | MessageQueue | LogicalQueue |
> > > |------------|--------------|--------------|
> > > | broker1    | 0            | 0            |
> > > | broker1    | 1            | 1            |
> > > | broker2    | 0            | 2            |
> > > | broker2    | 1            | 3            |
> > >
> > > After one LogicalQueue migrated from broker1 to broker2, there will be
> > two
> > > MessageQueues for one LogicalQueue. We only migrate mapping but not
> > actual
> > > data, so broker1 is still serving for old data consuming but not data
> > > producing.
> > >
> > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > |------------|--------------|--------------|--------------------|
> > > | **broker1**    | **0**            | **0(0-100)**         |
> **ReadOnly**
> > >         |
> > > | broker1    | 1            | 1            | Normal             |
> > > | broker2    | 0            | 2            | Normal             |
> > > | broker2    | 1            | 3            | Normal             |
> > > | **broker2**    | **2**            | **0(101-)**         | **Normal**
> > >         |
> > >
> > > After broker1 cleans all data from the commit log and consume queue,
> > > QueueStatus becomes Expired(neither readable nor writable).
> > >
> > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > |------------|--------------|--------------|-------------|
> > > | **broker1**    | **0**            | **0(-)**         | **Expired**
> >  |
> > > | broker1    | 1            | 1            | Normal      |
> > > | broker2    | 0            | 2            | Normal      |
> > > | broker2    | 1            | 3            | Normal      |
> > > | broker2    | 2            | 0(101-)      | Normal      |
> > >
> > > If this LogicalQueue is migrated back to broker1, it will reuse this
> > > expired MessageQueue
> > >
> > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > |------------|--------------|--------------|-------------|
> > > | **broker1**    | **0**            | **0(201-)**      | **Normal**
> > |
> > > | broker1    | 1            | 1            | Normal      |
> > > | broker2    | 0            | 2            | Normal      |
> > > | broker2    | 1            | 3            | Normal      |
> > > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**
> > |
> > >
> > > If this LogicalQueue is migrated back to broker1 while MessageQueue not
> > > expired, it will create a new MessageQueue
> > >
> > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > |------------|--------------|--------------|-------------|
> > > | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**
> > |
> > > | broker1    | 1            | 1            | Normal      |
> > > | **broker1**    | **2**            | **0(201-)**      | **Normal**
> > |
> > > | broker2    | 0            | 2            | Normal      |
> > > | broker2    | 1            | 3            | Normal      |
> > > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**
> > |
> > >
> > > If broker2 is offlined, all LogicalQueue in this broker should be
> > migrated
> > > away.
> > >
> > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > |------------|--------------|--------------|-------------|
> > > | broker1    | 0            | 0            | Normal      |
> > > | broker1    | 1            | 1            | Normal      |
> > > | **broker1**    | **2**            | **2(101-)**      | **Normal**
> > |
> > > | **broker1**    | **3**            | **3(101-)**      | **Normal**
> > |
> > > | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**
> > |
> > > | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**
> > |
> > >
> > > When all data including commit log and consume queue in broker2 are
> > > cleaned, broker2 can be removed.
> > >
> > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > |------------|--------------|--------------|-------------|
> > > | broker1    | 0            | 0            | Normal      |
> > > | broker1    | 1            | 1            | Normal      |
> > > | **broker1**    | **2**            | **2(101-)**     | **Normal**
>   |
> > > | **broker1**    | **3**            | **3(101-)**      | **Normal**
> > |
> > >
> > > When a new broker is deployed, we can migrate some LogicalQueues to
> this
> > > broker to spare some producing traffic.
> > >
> > > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > > |------------|--------------|--------------|-------------|
> > > | broker1    | 0            | 0            | Normal      |
> > > | broker1    | 1            | 1            | Normal      |
> > > | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**
> > |
> > > | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**
> > |
> > > | **broker3**    | **0**            | **2(201-)**      | **Normal**
> > |
> > > | **broker3**    | **1**            | **3(201-)**      | **Normal**
> > |
> > >
> > > All mapping data are stored separately in brokers, and via
> registerBroker
> > > to be gathered in namesrv; Each broker only stores its own logical
> queue
> > > mapping but not other one's. All management operations should be
> invoked
> > by
> > > CLI or direct rpc command, automatic operation support is missing now,
> > and
> > > require [RIP-18](
> > >
> > >
> >
> https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
> > > )
> > > to be implemented first.
> > >
> > > ## Interface Design/Change
> > > - Method signature changes
> > > No.
> > > - Method behavior changes
> > > When a topic is enabled LogicalQueue support, broker name of
> MessageQueue
> > > result returned by some methods like `fetchSubscribeMessageQueues` or
> > > `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does
> > not
> > > have broker name concept.
> > > - CLI command changes
> > > Add some operation command for LogicalQueue, like
> > > `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
> > > `QueryLogicalQueueMapping` `UpdateLogicalQueueNum`
> `MigrateLogicalQueue`.
> > > Also `updateTopic` adds a new parameter to enable LogicalQueue support
> > > immediately after the topic is created.
> > > - Log format or content changes
> > > No.
> > > ## Compatibility, Deprecation, and Migration Plan
> > > - Are backward and forward compatibility taken into consideration?
> > > Yes.
> > > - Everything will work well if no topic is enabled LogicalQueue
> support,
> > > whether on old/new broker/namesrv/client.
> > > - LogicalQueue support will work only under new broker+namesrv+client,
> > > otherwise, everything will work like no LogicalQueue support.
> > > - Are there deprecated APIs?
> > > No.
> > > - How do we do migration?
> > > No need to migrate, this is a feature which needs to be enabled
> manually.
> > > ## Implementation Outline
> > > We will implement the proposed changes by 2 phases.
> > > ### Phase 1 basic function
> > > 1. Implement LogicalQueue routes query and merge mechanism in namesrv.
> > > 2. Implement LogicalQueue mapping report from broker to namesrv.
> > > 3. Implement LogicalQueue support in the client.
> > > 4. Implement LogicalQueue migration in broker.
> > > 5. Implement LogicalQueue admin processor and CLI.
> > > ### Phase 2 compatibility
> > > 1. Implement proxy request+response for old client in broker.
> > > 2. Implement intermediate state(support is partly enabled in some
> brokers
> > > but not all) protection in namesrv.
> > > # Rejected Alternatives
> > > - How does alternatives solve the issue you proposed?
> > > Implement a whole new Logical Queue architecture from scratch, this
> > > absolutely will solve the problem.
> > > - Pros and Cons of alternatives
> > > Pros: from-scratch way does not bring anything good.
> > > Cons: from-scratch way will break existent concept, add much more
> > > complexity to code and not user-friendly.
> > > - Why should we reject the above alternatives
> > > It does no good.
> > >
> >
> #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
> This e-mail and its attachments contain confidential information from
> XIAOMI, which is intended only for the person or entity whose address is
> listed above. Any use of the information contained herein in any way
> (including, but not limited to, total or partial disclosure, reproduction,
> or dissemination) by persons other than the intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!******/#
>

答复: [External Mail]Re: [VOTE] RIP-21 RocketMQ Logical Queue

Posted by Xu16 Zhang 张旭 <zh...@xiaomi.com.INVALID>.
+1



--------
张旭
人工智能与云平台-云技术-计算平台组
电话: 15011056041

________________________________________
发件人: zhibo li <os...@gmail.com>
发送时间: 2021年5月31日 15:00
收件人: dev@rocketmq.apache.org
主题: [External Mail]Re: [VOTE] RIP-21 RocketMQ Logical Queue

*外部邮件，谨慎处理 | This message originated from outside of XIAOMI. Please treat this email with caution*


+1

heng du <du...@apache.org> 于2021年5月29日周六 上午9:55写道：

> Hi RocketMQ Community,
>
> This is the vote for the kickoff of RIP-21 RocketMQ logic queue.
>
> In order to better meet the needs of stream computing and scaling, the
> concept of logical queues is proposed in this RIP, which enables RocketMQ
> to have the ability to scale in seconds without changing the number of
> queues.
>
> The vote will be open for at least 72 hours or until a necessary number
> of votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
>
> ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：
>
> > Below is Markdown text with some GFM syntax.
> >
> > # Status
> > - Current State: Proposed
> > - Authors: [ayanamist](https://github.com/ayanamist)
> > - Shepherds: duhengforever@apache.org
> > - Mailing List discussion: dev@rocketmq.apache.org
> > - Pull Request: #PR_NUMBER
> > - Released: <relased_version>
> > # Background & Motivation
> > What do we need to do
> > - Will we add a new module?
> > No.
> > - Will we add new APIs?
> > It will add a new constant to mock broker names for logical queues.
> > - Will we add a new feature?
> > Yes.
> >
> > Why should we do that
> > - Are there any problems with our current project?
> > Currently, the MessageQueue of RocketMQ is coupled with broker name,
> which
> > results that the queue number will change if broker number increases or
> > decreases, which causes all queues to rebalance, which may cause service
> > disruption like flink job restarts in minutes.
> > - What can we benefit from proposed changes?
> > The number of logical queues is not related with the number of brokers:
> We
> > can increase broker number without changing logical queue number,
> moreover,
> > we can increase logical queue number without deploying a new broker.
> > # Goals
> > - What problem is this proposal designed to solve?
> > Provide an abstraction, logical queue, to decouple between queue number
> and
> > broker number.
> > - To what degree should we solve the problem?
> > We should not hurt availability or performance in the implementation.
> > # Non-Goals
> > - What problem is this proposal NOT designed to solve?
> > We will not improve the mechanism of queues rebalance.
> > - Are there any limits of this proposal?
> > Only newer clients with changes in this proposal will benefit.
> > # Changes
> > ## Architecture
> >
> > We use one or more MessageQueue to make one LogicalQueue; One
> LogicalQueue
> > is unique in one topic, and one MessageQueue belongs to one and only one
> > LogicalQueue.
> >
> > | brokerName | MessageQueue | LogicalQueue |
> > |------------|--------------|--------------|
> > | broker1    | 0            | 0            |
> > | broker1    | 1            | 1            |
> > | broker2    | 0            | 2            |
> > | broker2    | 1            | 3            |
> >
> > After one LogicalQueue migrated from broker1 to broker2, there will be
> two
> > MessageQueues for one LogicalQueue. We only migrate mapping but not
> actual
> > data, so broker1 is still serving for old data consuming but not data
> > producing.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|--------------------|
> > | **broker1**    | **0**            | **0(0-100)**         | **ReadOnly**
> >         |
> > | broker1    | 1            | 1            | Normal             |
> > | broker2    | 0            | 2            | Normal             |
> > | broker2    | 1            | 3            | Normal             |
> > | **broker2**    | **2**            | **0(101-)**         | **Normal**
> >         |
> >
> > After broker1 cleans all data from the commit log and consume queue,
> > QueueStatus becomes Expired(neither readable nor writable).
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(-)**         | **Expired**
>  |
> > | broker1    | 1            | 1            | Normal      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | broker2    | 2            | 0(101-)      | Normal      |
> >
> > If this LogicalQueue is migrated back to broker1, it will reuse this
> > expired MessageQueue
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(201-)**      | **Normal**
> |
> > | broker1    | 1            | 1            | Normal      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**
> |
> >
> > If this LogicalQueue is migrated back to broker1 while MessageQueue not
> > expired, it will create a new MessageQueue
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**
> |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **0(201-)**      | **Normal**
> |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**
> |
> >
> > If broker2 is offlined, all LogicalQueue in this broker should be
> migrated
> > away.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-)**      | **Normal**
> |
> > | **broker1**    | **3**            | **3(101-)**      | **Normal**
> |
> > | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**
> |
> > | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**
> |
> >
> > When all data including commit log and consume queue in broker2 are
> > cleaned, broker2 can be removed.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-)**     | **Normal**      |
> > | **broker1**    | **3**            | **3(101-)**      | **Normal**
> |
> >
> > When a new broker is deployed, we can migrate some LogicalQueues to this
> > broker to spare some producing traffic.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**
> |
> > | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**
> |
> > | **broker3**    | **0**            | **2(201-)**      | **Normal**
> |
> > | **broker3**    | **1**            | **3(201-)**      | **Normal**
> |
> >
> > All mapping data are stored separately in brokers, and via registerBroker
> > to be gathered in namesrv; Each broker only stores its own logical queue
> > mapping but not other one's. All management operations should be invoked
> by
> > CLI or direct rpc command, automatic operation support is missing now,
> and
> > require [RIP-18](
> >
> >
> https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
> > )
> > to be implemented first.
> >
> > ## Interface Design/Change
> > - Method signature changes
> > No.
> > - Method behavior changes
> > When a topic is enabled LogicalQueue support, broker name of MessageQueue
> > result returned by some methods like `fetchSubscribeMessageQueues` or
> > `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does
> not
> > have broker name concept.
> > - CLI command changes
> > Add some operation command for LogicalQueue, like
> > `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
> > `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`.
> > Also `updateTopic` adds a new parameter to enable LogicalQueue support
> > immediately after the topic is created.
> > - Log format or content changes
> > No.
> > ## Compatibility, Deprecation, and Migration Plan
> > - Are backward and forward compatibility taken into consideration?
> > Yes.
> > - Everything will work well if no topic is enabled LogicalQueue support,
> > whether on old/new broker/namesrv/client.
> > - LogicalQueue support will work only under new broker+namesrv+client,
> > otherwise, everything will work like no LogicalQueue support.
> > - Are there deprecated APIs?
> > No.
> > - How do we do migration?
> > No need to migrate, this is a feature which needs to be enabled manually.
> > ## Implementation Outline
> > We will implement the proposed changes by 2 phases.
> > ### Phase 1 basic function
> > 1. Implement LogicalQueue routes query and merge mechanism in namesrv.
> > 2. Implement LogicalQueue mapping report from broker to namesrv.
> > 3. Implement LogicalQueue support in the client.
> > 4. Implement LogicalQueue migration in broker.
> > 5. Implement LogicalQueue admin processor and CLI.
> > ### Phase 2 compatibility
> > 1. Implement proxy request+response for old client in broker.
> > 2. Implement intermediate state(support is partly enabled in some brokers
> > but not all) protection in namesrv.
> > # Rejected Alternatives
> > - How does alternatives solve the issue you proposed?
> > Implement a whole new Logical Queue architecture from scratch, this
> > absolutely will solve the problem.
> > - Pros and Cons of alternatives
> > Pros: from-scratch way does not bring anything good.
> > Cons: from-scratch way will break existent concept, add much more
> > complexity to code and not user-friendly.
> > - Why should we reject the above alternatives
> > It does no good.
> >
>
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#

Re: [VOTE] RIP-21 RocketMQ Logical Queue

Posted by zhibo li <os...@gmail.com>.
+1

heng du <du...@apache.org> 于2021年5月29日周六 上午9:55写道：

> Hi RocketMQ Community,
>
> This is the vote for the kickoff of RIP-21 RocketMQ logic queue.
>
> In order to better meet the needs of stream computing and scaling, the
> concept of logical queues is proposed in this RIP, which enables RocketMQ
> to have the ability to scale in seconds without changing the number of
> queues.
>
> The vote will be open for at least 72 hours or until a necessary number
> of votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
>
> ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：
>
> > Below is Markdown text with some GFM syntax.
> >
> > # Status
> > - Current State: Proposed
> > - Authors: [ayanamist](https://github.com/ayanamist)
> > - Shepherds: duhengforever@apache.org
> > - Mailing List discussion: dev@rocketmq.apache.org
> > - Pull Request: #PR_NUMBER
> > - Released: <relased_version>
> > # Background & Motivation
> > What do we need to do
> > - Will we add a new module?
> > No.
> > - Will we add new APIs?
> > It will add a new constant to mock broker names for logical queues.
> > - Will we add a new feature?
> > Yes.
> >
> > Why should we do that
> > - Are there any problems with our current project?
> > Currently, the MessageQueue of RocketMQ is coupled with broker name,
> which
> > results that the queue number will change if broker number increases or
> > decreases, which causes all queues to rebalance, which may cause service
> > disruption like flink job restarts in minutes.
> > - What can we benefit from proposed changes?
> > The number of logical queues is not related with the number of brokers:
> We
> > can increase broker number without changing logical queue number,
> moreover,
> > we can increase logical queue number without deploying a new broker.
> > # Goals
> > - What problem is this proposal designed to solve?
> > Provide an abstraction, logical queue, to decouple between queue number
> and
> > broker number.
> > - To what degree should we solve the problem?
> > We should not hurt availability or performance in the implementation.
> > # Non-Goals
> > - What problem is this proposal NOT designed to solve?
> > We will not improve the mechanism of queues rebalance.
> > - Are there any limits of this proposal?
> > Only newer clients with changes in this proposal will benefit.
> > # Changes
> > ## Architecture
> >
> > We use one or more MessageQueue to make one LogicalQueue; One
> LogicalQueue
> > is unique in one topic, and one MessageQueue belongs to one and only one
> > LogicalQueue.
> >
> > | brokerName | MessageQueue | LogicalQueue |
> > |------------|--------------|--------------|
> > | broker1    | 0            | 0            |
> > | broker1    | 1            | 1            |
> > | broker2    | 0            | 2            |
> > | broker2    | 1            | 3            |
> >
> > After one LogicalQueue migrated from broker1 to broker2, there will be
> two
> > MessageQueues for one LogicalQueue. We only migrate mapping but not
> actual
> > data, so broker1 is still serving for old data consuming but not data
> > producing.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|--------------------|
> > | **broker1**    | **0**            | **0(0-100)**         | **ReadOnly**
> >         |
> > | broker1    | 1            | 1            | Normal             |
> > | broker2    | 0            | 2            | Normal             |
> > | broker2    | 1            | 3            | Normal             |
> > | **broker2**    | **2**            | **0(101-)**         | **Normal**
> >         |
> >
> > After broker1 cleans all data from the commit log and consume queue,
> > QueueStatus becomes Expired(neither readable nor writable).
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(-)**         | **Expired**
>  |
> > | broker1    | 1            | 1            | Normal      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | broker2    | 2            | 0(101-)      | Normal      |
> >
> > If this LogicalQueue is migrated back to broker1, it will reuse this
> > expired MessageQueue
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(201-)**      | **Normal**
> |
> > | broker1    | 1            | 1            | Normal      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**
> |
> >
> > If this LogicalQueue is migrated back to broker1 while MessageQueue not
> > expired, it will create a new MessageQueue
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**
> |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **0(201-)**      | **Normal**
> |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**
> |
> >
> > If broker2 is offlined, all LogicalQueue in this broker should be
> migrated
> > away.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-)**      | **Normal**
> |
> > | **broker1**    | **3**            | **3(101-)**      | **Normal**
> |
> > | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**
> |
> > | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**
> |
> >
> > When all data including commit log and consume queue in broker2 are
> > cleaned, broker2 can be removed.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-)**     | **Normal**      |
> > | **broker1**    | **3**            | **3(101-)**      | **Normal**
> |
> >
> > When a new broker is deployed, we can migrate some LogicalQueues to this
> > broker to spare some producing traffic.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**
> |
> > | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**
> |
> > | **broker3**    | **0**            | **2(201-)**      | **Normal**
> |
> > | **broker3**    | **1**            | **3(201-)**      | **Normal**
> |
> >
> > All mapping data are stored separately in brokers, and via registerBroker
> > to be gathered in namesrv; Each broker only stores its own logical queue
> > mapping but not other one's. All management operations should be invoked
> by
> > CLI or direct rpc command, automatic operation support is missing now,
> and
> > require [RIP-18](
> >
> >
> https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
> > )
> > to be implemented first.
> >
> > ## Interface Design/Change
> > - Method signature changes
> > No.
> > - Method behavior changes
> > When a topic is enabled LogicalQueue support, broker name of MessageQueue
> > result returned by some methods like `fetchSubscribeMessageQueues` or
> > `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does
> not
> > have broker name concept.
> > - CLI command changes
> > Add some operation command for LogicalQueue, like
> > `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
> > `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`.
> > Also `updateTopic` adds a new parameter to enable LogicalQueue support
> > immediately after the topic is created.
> > - Log format or content changes
> > No.
> > ## Compatibility, Deprecation, and Migration Plan
> > - Are backward and forward compatibility taken into consideration?
> > Yes.
> > - Everything will work well if no topic is enabled LogicalQueue support,
> > whether on old/new broker/namesrv/client.
> > - LogicalQueue support will work only under new broker+namesrv+client,
> > otherwise, everything will work like no LogicalQueue support.
> > - Are there deprecated APIs?
> > No.
> > - How do we do migration?
> > No need to migrate, this is a feature which needs to be enabled manually.
> > ## Implementation Outline
> > We will implement the proposed changes by 2 phases.
> > ### Phase 1 basic function
> > 1. Implement LogicalQueue routes query and merge mechanism in namesrv.
> > 2. Implement LogicalQueue mapping report from broker to namesrv.
> > 3. Implement LogicalQueue support in the client.
> > 4. Implement LogicalQueue migration in broker.
> > 5. Implement LogicalQueue admin processor and CLI.
> > ### Phase 2 compatibility
> > 1. Implement proxy request+response for old client in broker.
> > 2. Implement intermediate state(support is partly enabled in some brokers
> > but not all) protection in namesrv.
> > # Rejected Alternatives
> > - How does alternatives solve the issue you proposed?
> > Implement a whole new Logical Queue architecture from scratch, this
> > absolutely will solve the problem.
> > - Pros and Cons of alternatives
> > Pros: from-scratch way does not bring anything good.
> > Cons: from-scratch way will break existent concept, add much more
> > complexity to code and not user-friendly.
> > - Why should we reject the above alternatives
> > It does no good.
> >
>

[RESULT][VOTE] RIP-21 RocketMQ Logical Queue

Posted by heng du <du...@apache.org>.
Hello RocketMQ Community,

This is the vote result for the kickoff of RIP-21 Apache RocketMQ logic
queue support, and it has been passed with [3] binding +1s, [3] non-binding:

*Binding votes +1s:*

heng du (duhengforever)

dongeforever(dongeforever)

Rongtongjin(Jinrongtong)


*Non-binding votes +1s:*

zhiboli
xu16 Zhang
炼龙




This RIP will be accepted and its status will be updated to RocketMQ  Wiki
soon.

Thanks,
The Apache RocketMQ Team

heng du <du...@apache.org> 于2021年5月29日周六 上午9:55写道：

> Hi RocketMQ Community,
>
> This is the vote for the kickoff of RIP-21 RocketMQ logic queue.
>
> In order to better meet the needs of stream computing and scaling, the
> concept of logical queues is proposed in this RIP, which enables RocketMQ
> to have the ability to scale in seconds without changing the number of
> queues.
>
> The vote will be open for at least 72 hours or until a necessary number
> of votes are reached.
>
> Please vote accordingly:
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
>
>
> ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：
>
>> Below is Markdown text with some GFM syntax.
>>
>> # Status
>> - Current State: Proposed
>> - Authors: [ayanamist](https://github.com/ayanamist)
>> - Shepherds: duhengforever@apache.org
>> - Mailing List discussion: dev@rocketmq.apache.org
>> - Pull Request: #PR_NUMBER
>> - Released: <relased_version>
>> # Background & Motivation
>> What do we need to do
>> - Will we add a new module?
>> No.
>> - Will we add new APIs?
>> It will add a new constant to mock broker names for logical queues.
>> - Will we add a new feature?
>> Yes.
>>
>> Why should we do that
>> - Are there any problems with our current project?
>> Currently, the MessageQueue of RocketMQ is coupled with broker name, which
>> results that the queue number will change if broker number increases or
>> decreases, which causes all queues to rebalance, which may cause service
>> disruption like flink job restarts in minutes.
>> - What can we benefit from proposed changes?
>> The number of logical queues is not related with the number of brokers: We
>> can increase broker number without changing logical queue number,
>> moreover,
>> we can increase logical queue number without deploying a new broker.
>> # Goals
>> - What problem is this proposal designed to solve?
>> Provide an abstraction, logical queue, to decouple between queue number
>> and
>> broker number.
>> - To what degree should we solve the problem?
>> We should not hurt availability or performance in the implementation.
>> # Non-Goals
>> - What problem is this proposal NOT designed to solve?
>> We will not improve the mechanism of queues rebalance.
>> - Are there any limits of this proposal?
>> Only newer clients with changes in this proposal will benefit.
>> # Changes
>> ## Architecture
>>
>> We use one or more MessageQueue to make one LogicalQueue; One LogicalQueue
>> is unique in one topic, and one MessageQueue belongs to one and only one
>> LogicalQueue.
>>
>> | brokerName | MessageQueue | LogicalQueue |
>> |------------|--------------|--------------|
>> | broker1    | 0            | 0            |
>> | broker1    | 1            | 1            |
>> | broker2    | 0            | 2            |
>> | broker2    | 1            | 3            |
>>
>> After one LogicalQueue migrated from broker1 to broker2, there will be two
>> MessageQueues for one LogicalQueue. We only migrate mapping but not actual
>> data, so broker1 is still serving for old data consuming but not data
>> producing.
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|--------------------|
>> | **broker1**    | **0**            | **0(0-100)**         | **ReadOnly**
>>         |
>> | broker1    | 1            | 1            | Normal             |
>> | broker2    | 0            | 2            | Normal             |
>> | broker2    | 1            | 3            | Normal             |
>> | **broker2**    | **2**            | **0(101-)**         | **Normal**
>>         |
>>
>> After broker1 cleans all data from the commit log and consume queue,
>> QueueStatus becomes Expired(neither readable nor writable).
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | **broker1**    | **0**            | **0(-)**         | **Expired**     |
>> | broker1    | 1            | 1            | Normal      |
>> | broker2    | 0            | 2            | Normal      |
>> | broker2    | 1            | 3            | Normal      |
>> | broker2    | 2            | 0(101-)      | Normal      |
>>
>> If this LogicalQueue is migrated back to broker1, it will reuse this
>> expired MessageQueue
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | **broker1**    | **0**            | **0(201-)**      | **Normal**      |
>> | broker1    | 1            | 1            | Normal      |
>> | broker2    | 0            | 2            | Normal      |
>> | broker2    | 1            | 3            | Normal      |
>> | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
>>
>> If this LogicalQueue is migrated back to broker1 while MessageQueue not
>> expired, it will create a new MessageQueue
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**    |
>> | broker1    | 1            | 1            | Normal      |
>> | **broker1**    | **2**            | **0(201-)**      | **Normal**      |
>> | broker2    | 0            | 2            | Normal      |
>> | broker2    | 1            | 3            | Normal      |
>> | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
>>
>> If broker2 is offlined, all LogicalQueue in this broker should be migrated
>> away.
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | broker1    | 0            | 0            | Normal      |
>> | broker1    | 1            | 1            | Normal      |
>> | **broker1**    | **2**            | **2(101-)**      | **Normal**      |
>> | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
>> | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**    |
>> | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**    |
>>
>> When all data including commit log and consume queue in broker2 are
>> cleaned, broker2 can be removed.
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | broker1    | 0            | 0            | Normal      |
>> | broker1    | 1            | 1            | Normal      |
>> | **broker1**    | **2**            | **2(101-)**     | **Normal**      |
>> | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
>>
>> When a new broker is deployed, we can migrate some LogicalQueues to this
>> broker to spare some producing traffic.
>>
>> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
>> |------------|--------------|--------------|-------------|
>> | broker1    | 0            | 0            | Normal      |
>> | broker1    | 1            | 1            | Normal      |
>> | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**    |
>> | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**    |
>> | **broker3**    | **0**            | **2(201-)**      | **Normal**      |
>> | **broker3**    | **1**            | **3(201-)**      | **Normal**      |
>>
>> All mapping data are stored separately in brokers, and via registerBroker
>> to be gathered in namesrv; Each broker only stores its own logical queue
>> mapping but not other one's. All management operations should be invoked
>> by
>> CLI or direct rpc command, automatic operation support is missing now, and
>> require [RIP-18](
>>
>> https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
>> )
>> to be implemented first.
>>
>> ## Interface Design/Change
>> - Method signature changes
>> No.
>> - Method behavior changes
>> When a topic is enabled LogicalQueue support, broker name of MessageQueue
>> result returned by some methods like `fetchSubscribeMessageQueues` or
>> `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does
>> not
>> have broker name concept.
>> - CLI command changes
>> Add some operation command for LogicalQueue, like
>> `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
>> `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`.
>> Also `updateTopic` adds a new parameter to enable LogicalQueue support
>> immediately after the topic is created.
>> - Log format or content changes
>> No.
>> ## Compatibility, Deprecation, and Migration Plan
>> - Are backward and forward compatibility taken into consideration?
>> Yes.
>> - Everything will work well if no topic is enabled LogicalQueue support,
>> whether on old/new broker/namesrv/client.
>> - LogicalQueue support will work only under new broker+namesrv+client,
>> otherwise, everything will work like no LogicalQueue support.
>> - Are there deprecated APIs?
>> No.
>> - How do we do migration?
>> No need to migrate, this is a feature which needs to be enabled manually.
>> ## Implementation Outline
>> We will implement the proposed changes by 2 phases.
>> ### Phase 1 basic function
>> 1. Implement LogicalQueue routes query and merge mechanism in namesrv.
>> 2. Implement LogicalQueue mapping report from broker to namesrv.
>> 3. Implement LogicalQueue support in the client.
>> 4. Implement LogicalQueue migration in broker.
>> 5. Implement LogicalQueue admin processor and CLI.
>> ### Phase 2 compatibility
>> 1. Implement proxy request+response for old client in broker.
>> 2. Implement intermediate state(support is partly enabled in some brokers
>> but not all) protection in namesrv.
>> # Rejected Alternatives
>> - How does alternatives solve the issue you proposed?
>> Implement a whole new Logical Queue architecture from scratch, this
>> absolutely will solve the problem.
>> - Pros and Cons of alternatives
>> Pros: from-scratch way does not bring anything good.
>> Cons: from-scratch way will break existent concept, add much more
>> complexity to code and not user-friendly.
>> - Why should we reject the above alternatives
>> It does no good.
>>
>

Re: [VOTE] RIP-21 RocketMQ Logical Queue

Posted by Rongtong Jin <ji...@mails.ucas.ac.cn>.
+1

&quot;heng du&quot; &lt;duhengforever@apache.org&gt;写道：
> Hi RocketMQ Community,
> 
> This is the vote for the kickoff of RIP-21 RocketMQ logic queue.
> 
> In order to better meet the needs of stream computing and scaling, the
> concept of logical queues is proposed in this RIP, which enables RocketMQ
> to have the ability to scale in seconds without changing the number of
> queues.
> 
> The vote will be open for at least 72 hours or until a necessary number
> of votes are reached.
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> 
> 
> ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：
> 
> > Below is Markdown text with some GFM syntax.
> >
> > # Status
> > - Current State: Proposed
> > - Authors: [ayanamist](https://github.com/ayanamist)
> > - Shepherds: duhengforever@apache.org
> > - Mailing List discussion: dev@rocketmq.apache.org
> > - Pull Request: #PR_NUMBER
> > - Released: <relased_version>
> > # Background & Motivation
> > What do we need to do
> > - Will we add a new module?
> > No.
> > - Will we add new APIs?
> > It will add a new constant to mock broker names for logical queues.
> > - Will we add a new feature?
> > Yes.
> >
> > Why should we do that
> > - Are there any problems with our current project?
> > Currently, the MessageQueue of RocketMQ is coupled with broker name, which
> > results that the queue number will change if broker number increases or
> > decreases, which causes all queues to rebalance, which may cause service
> > disruption like flink job restarts in minutes.
> > - What can we benefit from proposed changes?
> > The number of logical queues is not related with the number of brokers: We
> > can increase broker number without changing logical queue number, moreover,
> > we can increase logical queue number without deploying a new broker.
> > # Goals
> > - What problem is this proposal designed to solve?
> > Provide an abstraction, logical queue, to decouple between queue number and
> > broker number.
> > - To what degree should we solve the problem?
> > We should not hurt availability or performance in the implementation.
> > # Non-Goals
> > - What problem is this proposal NOT designed to solve?
> > We will not improve the mechanism of queues rebalance.
> > - Are there any limits of this proposal?
> > Only newer clients with changes in this proposal will benefit.
> > # Changes
> > ## Architecture
> >
> > We use one or more MessageQueue to make one LogicalQueue; One LogicalQueue
> > is unique in one topic, and one MessageQueue belongs to one and only one
> > LogicalQueue.
> >
> > | brokerName | MessageQueue | LogicalQueue |
> > |------------|--------------|--------------|
> > | broker1    | 0            | 0            |
> > | broker1    | 1            | 1            |
> > | broker2    | 0            | 2            |
> > | broker2    | 1            | 3            |
> >
> > After one LogicalQueue migrated from broker1 to broker2, there will be two
> > MessageQueues for one LogicalQueue. We only migrate mapping but not actual
> > data, so broker1 is still serving for old data consuming but not data
> > producing.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|--------------------|
> > | **broker1**    | **0**            | **0(0-100)**         | **ReadOnly**
> >         |
> > | broker1    | 1            | 1            | Normal             |
> > | broker2    | 0            | 2            | Normal             |
> > | broker2    | 1            | 3            | Normal             |
> > | **broker2**    | **2**            | **0(101-)**         | **Normal**
> >         |
> >
> > After broker1 cleans all data from the commit log and consume queue,
> > QueueStatus becomes Expired(neither readable nor writable).
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(-)**         | **Expired**     |
> > | broker1    | 1            | 1            | Normal      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | broker2    | 2            | 0(101-)      | Normal      |
> >
> > If this LogicalQueue is migrated back to broker1, it will reuse this
> > expired MessageQueue
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(201-)**      | **Normal**      |
> > | broker1    | 1            | 1            | Normal      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
> >
> > If this LogicalQueue is migrated back to broker1 while MessageQueue not
> > expired, it will create a new MessageQueue
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**    |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **0(201-)**      | **Normal**      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
> >
> > If broker2 is offlined, all LogicalQueue in this broker should be migrated
> > away.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-)**      | **Normal**      |
> > | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
> > | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**    |
> > | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**    |
> >
> > When all data including commit log and consume queue in broker2 are
> > cleaned, broker2 can be removed.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-)**     | **Normal**      |
> > | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
> >
> > When a new broker is deployed, we can migrate some LogicalQueues to this
> > broker to spare some producing traffic.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**    |
> > | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**    |
> > | **broker3**    | **0**            | **2(201-)**      | **Normal**      |
> > | **broker3**    | **1**            | **3(201-)**      | **Normal**      |
> >
> > All mapping data are stored separately in brokers, and via registerBroker
> > to be gathered in namesrv; Each broker only stores its own logical queue
> > mapping but not other one's. All management operations should be invoked by
> > CLI or direct rpc command, automatic operation support is missing now, and
> > require [RIP-18](
> >
> > https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
> > )
> > to be implemented first.
> >
> > ## Interface Design/Change
> > - Method signature changes
> > No.
> > - Method behavior changes
> > When a topic is enabled LogicalQueue support, broker name of MessageQueue
> > result returned by some methods like `fetchSubscribeMessageQueues` or
> > `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does not
> > have broker name concept.
> > - CLI command changes
> > Add some operation command for LogicalQueue, like
> > `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
> > `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`.
> > Also `updateTopic` adds a new parameter to enable LogicalQueue support
> > immediately after the topic is created.
> > - Log format or content changes
> > No.
> > ## Compatibility, Deprecation, and Migration Plan
> > - Are backward and forward compatibility taken into consideration?
> > Yes.
> > - Everything will work well if no topic is enabled LogicalQueue support,
> > whether on old/new broker/namesrv/client.
> > - LogicalQueue support will work only under new broker+namesrv+client,
> > otherwise, everything will work like no LogicalQueue support.
> > - Are there deprecated APIs?
> > No.
> > - How do we do migration?
> > No need to migrate, this is a feature which needs to be enabled manually.
> > ## Implementation Outline
> > We will implement the proposed changes by 2 phases.
> > ### Phase 1 basic function
> > 1. Implement LogicalQueue routes query and merge mechanism in namesrv.
> > 2. Implement LogicalQueue mapping report from broker to namesrv.
> > 3. Implement LogicalQueue support in the client.
> > 4. Implement LogicalQueue migration in broker.
> > 5. Implement LogicalQueue admin processor and CLI.
> > ### Phase 2 compatibility
> > 1. Implement proxy request+response for old client in broker.
> > 2. Implement intermediate state(support is partly enabled in some brokers
> > but not all) protection in namesrv.
> > # Rejected Alternatives
> > - How does alternatives solve the issue you proposed?
> > Implement a whole new Logical Queue architecture from scratch, this
> > absolutely will solve the problem.
> > - Pros and Cons of alternatives
> > Pros: from-scratch way does not bring anything good.
> > Cons: from-scratch way will break existent concept, add much more
> > complexity to code and not user-friendly.
> > - Why should we reject the above alternatives
> > It does no good.
> >

Re: [VOTE] RIP-21 RocketMQ Logical Queue

Posted by Rongtong Jin <ji...@mails.ucas.ac.cn>.
+1

&quot;heng du&quot; &lt;duhengforever@apache.org&gt;写道：
> Hi RocketMQ Community,
> 
> This is the vote for the kickoff of RIP-21 RocketMQ logic queue.
> 
> In order to better meet the needs of stream computing and scaling, the
> concept of logical queues is proposed in this RIP, which enables RocketMQ
> to have the ability to scale in seconds without changing the number of
> queues.
> 
> The vote will be open for at least 72 hours or until a necessary number
> of votes are reached.
> 
> Please vote accordingly:
> 
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
> 
> 
> 
> ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：
> 
> > Below is Markdown text with some GFM syntax.
> >
> > # Status
> > - Current State: Proposed
> > - Authors: [ayanamist](https://github.com/ayanamist)
> > - Shepherds: duhengforever@apache.org
> > - Mailing List discussion: dev@rocketmq.apache.org
> > - Pull Request: #PR_NUMBER
> > - Released: <relased_version>
> > # Background & Motivation
> > What do we need to do
> > - Will we add a new module?
> > No.
> > - Will we add new APIs?
> > It will add a new constant to mock broker names for logical queues.
> > - Will we add a new feature?
> > Yes.
> >
> > Why should we do that
> > - Are there any problems with our current project?
> > Currently, the MessageQueue of RocketMQ is coupled with broker name, which
> > results that the queue number will change if broker number increases or
> > decreases, which causes all queues to rebalance, which may cause service
> > disruption like flink job restarts in minutes.
> > - What can we benefit from proposed changes?
> > The number of logical queues is not related with the number of brokers: We
> > can increase broker number without changing logical queue number, moreover,
> > we can increase logical queue number without deploying a new broker.
> > # Goals
> > - What problem is this proposal designed to solve?
> > Provide an abstraction, logical queue, to decouple between queue number and
> > broker number.
> > - To what degree should we solve the problem?
> > We should not hurt availability or performance in the implementation.
> > # Non-Goals
> > - What problem is this proposal NOT designed to solve?
> > We will not improve the mechanism of queues rebalance.
> > - Are there any limits of this proposal?
> > Only newer clients with changes in this proposal will benefit.
> > # Changes
> > ## Architecture
> >
> > We use one or more MessageQueue to make one LogicalQueue; One LogicalQueue
> > is unique in one topic, and one MessageQueue belongs to one and only one
> > LogicalQueue.
> >
> > | brokerName | MessageQueue | LogicalQueue |
> > |------------|--------------|--------------|
> > | broker1    | 0            | 0            |
> > | broker1    | 1            | 1            |
> > | broker2    | 0            | 2            |
> > | broker2    | 1            | 3            |
> >
> > After one LogicalQueue migrated from broker1 to broker2, there will be two
> > MessageQueues for one LogicalQueue. We only migrate mapping but not actual
> > data, so broker1 is still serving for old data consuming but not data
> > producing.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|--------------------|
> > | **broker1**    | **0**            | **0(0-100)**         | **ReadOnly**
> >         |
> > | broker1    | 1            | 1            | Normal             |
> > | broker2    | 0            | 2            | Normal             |
> > | broker2    | 1            | 3            | Normal             |
> > | **broker2**    | **2**            | **0(101-)**         | **Normal**
> >         |
> >
> > After broker1 cleans all data from the commit log and consume queue,
> > QueueStatus becomes Expired(neither readable nor writable).
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(-)**         | **Expired**     |
> > | broker1    | 1            | 1            | Normal      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | broker2    | 2            | 0(101-)      | Normal      |
> >
> > If this LogicalQueue is migrated back to broker1, it will reuse this
> > expired MessageQueue
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(201-)**      | **Normal**      |
> > | broker1    | 1            | 1            | Normal      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
> >
> > If this LogicalQueue is migrated back to broker1 while MessageQueue not
> > expired, it will create a new MessageQueue
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**    |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **0(201-)**      | **Normal**      |
> > | broker2    | 0            | 2            | Normal      |
> > | broker2    | 1            | 3            | Normal      |
> > | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
> >
> > If broker2 is offlined, all LogicalQueue in this broker should be migrated
> > away.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-)**      | **Normal**      |
> > | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
> > | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**    |
> > | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**    |
> >
> > When all data including commit log and consume queue in broker2 are
> > cleaned, broker2 can be removed.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-)**     | **Normal**      |
> > | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
> >
> > When a new broker is deployed, we can migrate some LogicalQueues to this
> > broker to spare some producing traffic.
> >
> > | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> > |------------|--------------|--------------|-------------|
> > | broker1    | 0            | 0            | Normal      |
> > | broker1    | 1            | 1            | Normal      |
> > | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**    |
> > | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**    |
> > | **broker3**    | **0**            | **2(201-)**      | **Normal**      |
> > | **broker3**    | **1**            | **3(201-)**      | **Normal**      |
> >
> > All mapping data are stored separately in brokers, and via registerBroker
> > to be gathered in namesrv; Each broker only stores its own logical queue
> > mapping but not other one's. All management operations should be invoked by
> > CLI or direct rpc command, automatic operation support is missing now, and
> > require [RIP-18](
> >
> > https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
> > )
> > to be implemented first.
> >
> > ## Interface Design/Change
> > - Method signature changes
> > No.
> > - Method behavior changes
> > When a topic is enabled LogicalQueue support, broker name of MessageQueue
> > result returned by some methods like `fetchSubscribeMessageQueues` or
> > `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does not
> > have broker name concept.
> > - CLI command changes
> > Add some operation command for LogicalQueue, like
> > `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
> > `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`.
> > Also `updateTopic` adds a new parameter to enable LogicalQueue support
> > immediately after the topic is created.
> > - Log format or content changes
> > No.
> > ## Compatibility, Deprecation, and Migration Plan
> > - Are backward and forward compatibility taken into consideration?
> > Yes.
> > - Everything will work well if no topic is enabled LogicalQueue support,
> > whether on old/new broker/namesrv/client.
> > - LogicalQueue support will work only under new broker+namesrv+client,
> > otherwise, everything will work like no LogicalQueue support.
> > - Are there deprecated APIs?
> > No.
> > - How do we do migration?
> > No need to migrate, this is a feature which needs to be enabled manually.
> > ## Implementation Outline
> > We will implement the proposed changes by 2 phases.
> > ### Phase 1 basic function
> > 1. Implement LogicalQueue routes query and merge mechanism in namesrv.
> > 2. Implement LogicalQueue mapping report from broker to namesrv.
> > 3. Implement LogicalQueue support in the client.
> > 4. Implement LogicalQueue migration in broker.
> > 5. Implement LogicalQueue admin processor and CLI.
> > ### Phase 2 compatibility
> > 1. Implement proxy request+response for old client in broker.
> > 2. Implement intermediate state(support is partly enabled in some brokers
> > but not all) protection in namesrv.
> > # Rejected Alternatives
> > - How does alternatives solve the issue you proposed?
> > Implement a whole new Logical Queue architecture from scratch, this
> > absolutely will solve the problem.
> > - Pros and Cons of alternatives
> > Pros: from-scratch way does not bring anything good.
> > Cons: from-scratch way will break existent concept, add much more
> > complexity to code and not user-friendly.
> > - Why should we reject the above alternatives
> > It does no good.
> >

[VOTE] RIP-21 RocketMQ Logical Queue

Posted by heng du <du...@apache.org>.
Hi RocketMQ Community,

This is the vote for the kickoff of RIP-21 RocketMQ logic queue.

In order to better meet the needs of stream computing and scaling, the
concept of logical queues is proposed in this RIP, which enables RocketMQ
to have the ability to scale in seconds without changing the number of
queues.

The vote will be open for at least 72 hours or until a necessary number
of votes are reached.

Please vote accordingly:

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason



ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：

> Below is Markdown text with some GFM syntax.
>
> # Status
> - Current State: Proposed
> - Authors: [ayanamist](https://github.com/ayanamist)
> - Shepherds: duhengforever@apache.org
> - Mailing List discussion: dev@rocketmq.apache.org
> - Pull Request: #PR_NUMBER
> - Released: <relased_version>
> # Background & Motivation
> What do we need to do
> - Will we add a new module?
> No.
> - Will we add new APIs?
> It will add a new constant to mock broker names for logical queues.
> - Will we add a new feature?
> Yes.
>
> Why should we do that
> - Are there any problems with our current project?
> Currently, the MessageQueue of RocketMQ is coupled with broker name, which
> results that the queue number will change if broker number increases or
> decreases, which causes all queues to rebalance, which may cause service
> disruption like flink job restarts in minutes.
> - What can we benefit from proposed changes?
> The number of logical queues is not related with the number of brokers: We
> can increase broker number without changing logical queue number, moreover,
> we can increase logical queue number without deploying a new broker.
> # Goals
> - What problem is this proposal designed to solve?
> Provide an abstraction, logical queue, to decouple between queue number and
> broker number.
> - To what degree should we solve the problem?
> We should not hurt availability or performance in the implementation.
> # Non-Goals
> - What problem is this proposal NOT designed to solve?
> We will not improve the mechanism of queues rebalance.
> - Are there any limits of this proposal?
> Only newer clients with changes in this proposal will benefit.
> # Changes
> ## Architecture
>
> We use one or more MessageQueue to make one LogicalQueue; One LogicalQueue
> is unique in one topic, and one MessageQueue belongs to one and only one
> LogicalQueue.
>
> | brokerName | MessageQueue | LogicalQueue |
> |------------|--------------|--------------|
> | broker1    | 0            | 0            |
> | broker1    | 1            | 1            |
> | broker2    | 0            | 2            |
> | broker2    | 1            | 3            |
>
> After one LogicalQueue migrated from broker1 to broker2, there will be two
> MessageQueues for one LogicalQueue. We only migrate mapping but not actual
> data, so broker1 is still serving for old data consuming but not data
> producing.
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|--------------------|
> | **broker1**    | **0**            | **0(0-100)**         | **ReadOnly**
>         |
> | broker1    | 1            | 1            | Normal             |
> | broker2    | 0            | 2            | Normal             |
> | broker2    | 1            | 3            | Normal             |
> | **broker2**    | **2**            | **0(101-)**         | **Normal**
>         |
>
> After broker1 cleans all data from the commit log and consume queue,
> QueueStatus becomes Expired(neither readable nor writable).
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | **broker1**    | **0**            | **0(-)**         | **Expired**     |
> | broker1    | 1            | 1            | Normal      |
> | broker2    | 0            | 2            | Normal      |
> | broker2    | 1            | 3            | Normal      |
> | broker2    | 2            | 0(101-)      | Normal      |
>
> If this LogicalQueue is migrated back to broker1, it will reuse this
> expired MessageQueue
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | **broker1**    | **0**            | **0(201-)**      | **Normal**      |
> | broker1    | 1            | 1            | Normal      |
> | broker2    | 0            | 2            | Normal      |
> | broker2    | 1            | 3            | Normal      |
> | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
>
> If this LogicalQueue is migrated back to broker1 while MessageQueue not
> expired, it will create a new MessageQueue
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**    |
> | broker1    | 1            | 1            | Normal      |
> | **broker1**    | **2**            | **0(201-)**      | **Normal**      |
> | broker2    | 0            | 2            | Normal      |
> | broker2    | 1            | 3            | Normal      |
> | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
>
> If broker2 is offlined, all LogicalQueue in this broker should be migrated
> away.
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | broker1    | 0            | 0            | Normal      |
> | broker1    | 1            | 1            | Normal      |
> | **broker1**    | **2**            | **2(101-)**      | **Normal**      |
> | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
> | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**    |
> | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**    |
>
> When all data including commit log and consume queue in broker2 are
> cleaned, broker2 can be removed.
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | broker1    | 0            | 0            | Normal      |
> | broker1    | 1            | 1            | Normal      |
> | **broker1**    | **2**            | **2(101-)**     | **Normal**      |
> | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
>
> When a new broker is deployed, we can migrate some LogicalQueues to this
> broker to spare some producing traffic.
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | broker1    | 0            | 0            | Normal      |
> | broker1    | 1            | 1            | Normal      |
> | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**    |
> | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**    |
> | **broker3**    | **0**            | **2(201-)**      | **Normal**      |
> | **broker3**    | **1**            | **3(201-)**      | **Normal**      |
>
> All mapping data are stored separately in brokers, and via registerBroker
> to be gathered in namesrv; Each broker only stores its own logical queue
> mapping but not other one's. All management operations should be invoked by
> CLI or direct rpc command, automatic operation support is missing now, and
> require [RIP-18](
>
> https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
> )
> to be implemented first.
>
> ## Interface Design/Change
> - Method signature changes
> No.
> - Method behavior changes
> When a topic is enabled LogicalQueue support, broker name of MessageQueue
> result returned by some methods like `fetchSubscribeMessageQueues` or
> `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does not
> have broker name concept.
> - CLI command changes
> Add some operation command for LogicalQueue, like
> `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
> `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`.
> Also `updateTopic` adds a new parameter to enable LogicalQueue support
> immediately after the topic is created.
> - Log format or content changes
> No.
> ## Compatibility, Deprecation, and Migration Plan
> - Are backward and forward compatibility taken into consideration?
> Yes.
> - Everything will work well if no topic is enabled LogicalQueue support,
> whether on old/new broker/namesrv/client.
> - LogicalQueue support will work only under new broker+namesrv+client,
> otherwise, everything will work like no LogicalQueue support.
> - Are there deprecated APIs?
> No.
> - How do we do migration?
> No need to migrate, this is a feature which needs to be enabled manually.
> ## Implementation Outline
> We will implement the proposed changes by 2 phases.
> ### Phase 1 basic function
> 1. Implement LogicalQueue routes query and merge mechanism in namesrv.
> 2. Implement LogicalQueue mapping report from broker to namesrv.
> 3. Implement LogicalQueue support in the client.
> 4. Implement LogicalQueue migration in broker.
> 5. Implement LogicalQueue admin processor and CLI.
> ### Phase 2 compatibility
> 1. Implement proxy request+response for old client in broker.
> 2. Implement intermediate state(support is partly enabled in some brokers
> but not all) protection in namesrv.
> # Rejected Alternatives
> - How does alternatives solve the issue you proposed?
> Implement a whole new Logical Queue architecture from scratch, this
> absolutely will solve the problem.
> - Pros and Cons of alternatives
> Pros: from-scratch way does not bring anything good.
> Cons: from-scratch way will break existent concept, add much more
> complexity to code and not user-friendly.
> - Why should we reject the above alternatives
> It does no good.
>

[VOTE] RIP-21 RocketMQ Logical Queue

Posted by heng du <du...@apache.org>.
Hi RocketMQ Community,

This is the vote for the kickoff of RIP-21 RocketMQ logic queue.

In order to better meet the needs of stream computing and scaling, the
concept of logical queues is proposed in this RIP, which enables RocketMQ
to have the ability to scale in seconds without changing the number of
queues.

The vote will be open for at least 72 hours or until a necessary number
of votes are reached.

Please vote accordingly:

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason



ayanamist <ay...@gmail.com> 于2021年5月18日周二 上午10:14写道：

> Below is Markdown text with some GFM syntax.
>
> # Status
> - Current State: Proposed
> - Authors: [ayanamist](https://github.com/ayanamist)
> - Shepherds: duhengforever@apache.org
> - Mailing List discussion: dev@rocketmq.apache.org
> - Pull Request: #PR_NUMBER
> - Released: <relased_version>
> # Background & Motivation
> What do we need to do
> - Will we add a new module?
> No.
> - Will we add new APIs?
> It will add a new constant to mock broker names for logical queues.
> - Will we add a new feature?
> Yes.
>
> Why should we do that
> - Are there any problems with our current project?
> Currently, the MessageQueue of RocketMQ is coupled with broker name, which
> results that the queue number will change if broker number increases or
> decreases, which causes all queues to rebalance, which may cause service
> disruption like flink job restarts in minutes.
> - What can we benefit from proposed changes?
> The number of logical queues is not related with the number of brokers: We
> can increase broker number without changing logical queue number, moreover,
> we can increase logical queue number without deploying a new broker.
> # Goals
> - What problem is this proposal designed to solve?
> Provide an abstraction, logical queue, to decouple between queue number and
> broker number.
> - To what degree should we solve the problem?
> We should not hurt availability or performance in the implementation.
> # Non-Goals
> - What problem is this proposal NOT designed to solve?
> We will not improve the mechanism of queues rebalance.
> - Are there any limits of this proposal?
> Only newer clients with changes in this proposal will benefit.
> # Changes
> ## Architecture
>
> We use one or more MessageQueue to make one LogicalQueue; One LogicalQueue
> is unique in one topic, and one MessageQueue belongs to one and only one
> LogicalQueue.
>
> | brokerName | MessageQueue | LogicalQueue |
> |------------|--------------|--------------|
> | broker1    | 0            | 0            |
> | broker1    | 1            | 1            |
> | broker2    | 0            | 2            |
> | broker2    | 1            | 3            |
>
> After one LogicalQueue migrated from broker1 to broker2, there will be two
> MessageQueues for one LogicalQueue. We only migrate mapping but not actual
> data, so broker1 is still serving for old data consuming but not data
> producing.
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|--------------------|
> | **broker1**    | **0**            | **0(0-100)**         | **ReadOnly**
>         |
> | broker1    | 1            | 1            | Normal             |
> | broker2    | 0            | 2            | Normal             |
> | broker2    | 1            | 3            | Normal             |
> | **broker2**    | **2**            | **0(101-)**         | **Normal**
>         |
>
> After broker1 cleans all data from the commit log and consume queue,
> QueueStatus becomes Expired(neither readable nor writable).
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | **broker1**    | **0**            | **0(-)**         | **Expired**     |
> | broker1    | 1            | 1            | Normal      |
> | broker2    | 0            | 2            | Normal      |
> | broker2    | 1            | 3            | Normal      |
> | broker2    | 2            | 0(101-)      | Normal      |
>
> If this LogicalQueue is migrated back to broker1, it will reuse this
> expired MessageQueue
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | **broker1**    | **0**            | **0(201-)**      | **Normal**      |
> | broker1    | 1            | 1            | Normal      |
> | broker2    | 0            | 2            | Normal      |
> | broker2    | 1            | 3            | Normal      |
> | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
>
> If this LogicalQueue is migrated back to broker1 while MessageQueue not
> expired, it will create a new MessageQueue
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | **broker1**    | **0**            | **0(0-100)**     | **ReadOnly**    |
> | broker1    | 1            | 1            | Normal      |
> | **broker1**    | **2**            | **0(201-)**      | **Normal**      |
> | broker2    | 0            | 2            | Normal      |
> | broker2    | 1            | 3            | Normal      |
> | **broker2**    | **2**            | **0(101-200)**   | **ReadOnly**    |
>
> If broker2 is offlined, all LogicalQueue in this broker should be migrated
> away.
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | broker1    | 0            | 0            | Normal      |
> | broker1    | 1            | 1            | Normal      |
> | **broker1**    | **2**            | **2(101-)**      | **Normal**      |
> | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
> | **broker2**    | **0**            | **2(0-100)**     | **ReadOnly**    |
> | **broker2**    | **1**            | **3(0-100)**     | **ReadOnly**    |
>
> When all data including commit log and consume queue in broker2 are
> cleaned, broker2 can be removed.
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | broker1    | 0            | 0            | Normal      |
> | broker1    | 1            | 1            | Normal      |
> | **broker1**    | **2**            | **2(101-)**     | **Normal**      |
> | **broker1**    | **3**            | **3(101-)**      | **Normal**      |
>
> When a new broker is deployed, we can migrate some LogicalQueues to this
> broker to spare some producing traffic.
>
> | brokerName | MessageQueue | LogicalQueue | QueueStatus |
> |------------|--------------|--------------|-------------|
> | broker1    | 0            | 0            | Normal      |
> | broker1    | 1            | 1            | Normal      |
> | **broker1**    | **2**            | **2(101-200)**   | **ReadOnly**    |
> | **broker1**    | **3**            | **3(101-200)**   | **ReadOnly**    |
> | **broker3**    | **0**            | **2(201-)**      | **Normal**      |
> | **broker3**    | **1**            | **3(201-)**      | **Normal**      |
>
> All mapping data are stored separately in brokers, and via registerBroker
> to be gathered in namesrv; Each broker only stores its own logical queue
> mapping but not other one's. All management operations should be invoked by
> CLI or direct rpc command, automatic operation support is missing now, and
> require [RIP-18](
>
> https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade
> )
> to be implemented first.
>
> ## Interface Design/Change
> - Method signature changes
> No.
> - Method behavior changes
> When a topic is enabled LogicalQueue support, broker name of MessageQueue
> result returned by some methods like `fetchSubscribeMessageQueues` or
> `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does not
> have broker name concept.
> - CLI command changes
> Add some operation command for LogicalQueue, like
> `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping`
> `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`.
> Also `updateTopic` adds a new parameter to enable LogicalQueue support
> immediately after the topic is created.
> - Log format or content changes
> No.
> ## Compatibility, Deprecation, and Migration Plan
> - Are backward and forward compatibility taken into consideration?
> Yes.
> - Everything will work well if no topic is enabled LogicalQueue support,
> whether on old/new broker/namesrv/client.
> - LogicalQueue support will work only under new broker+namesrv+client,
> otherwise, everything will work like no LogicalQueue support.
> - Are there deprecated APIs?
> No.
> - How do we do migration?
> No need to migrate, this is a feature which needs to be enabled manually.
> ## Implementation Outline
> We will implement the proposed changes by 2 phases.
> ### Phase 1 basic function
> 1. Implement LogicalQueue routes query and merge mechanism in namesrv.
> 2. Implement LogicalQueue mapping report from broker to namesrv.
> 3. Implement LogicalQueue support in the client.
> 4. Implement LogicalQueue migration in broker.
> 5. Implement LogicalQueue admin processor and CLI.
> ### Phase 2 compatibility
> 1. Implement proxy request+response for old client in broker.
> 2. Implement intermediate state(support is partly enabled in some brokers
> but not all) protection in namesrv.
> # Rejected Alternatives
> - How does alternatives solve the issue you proposed?
> Implement a whole new Logical Queue architecture from scratch, this
> absolutely will solve the problem.
> - Pros and Cons of alternatives
> Pros: from-scratch way does not bring anything good.
> Cons: from-scratch way will break existent concept, add much more
> complexity to code and not user-friendly.
> - Why should we reject the above alternatives
> It does no good.
>