You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by "huzongtang@cmss.chinamobile.com" <hu...@cmss.chinamobile.com> on 2019/03/14 06:31:14 UTC
[DISCUSS][RIP-12]Message Connector
Hi RocketMQ Community,
This is a discuss for a new feature which called Message Connector.The content of this RIP is as following.You can look at it.If you have any question about this rip,you can discuss it with me in the email.
1、Status
Current State: Discuss
Authors: HuZongTang, WangShaoJie
Shepherds: DuHeng
Mailing List discussion: users@rocketmq.apache.org;dev@rocketmq.apache.org
Pull Request:
Released: <5.0.0>
2、Background & Motivation
What do we need to do
In current version of rocketmq project, different cluster can't exchange messages each other. Although dLedger structure improves availability of rocketmq much more, it still can't implement backuping messages for different cluster.
So, we need a mechanism to implement backuping messages for different cluster. It can improve availability of multiple rocketmq cluster. Here, we called it Message Connector.
3、Goals
What problem is this proposal designed to solve?
We plan to design a component call Message Connector for RocketMQ. User can define the topic message routing rule by configing file, such as source topic, targe topic, filter tag , source cluster and targe cluster info.
To what degree should we solve the problem?
(1)Define some rules in configing files;
(2)Better availability for messages in multiple cluster enviroment.
(3)Monitering some parameters in the process of message replication;
4、Non-Goals
What problem is this proposal NOT designed to solve?
In this phase, this rip only support starting message replication from the latest message position currently in the message queue.
Are there any limits to this proposal?
If users want to use this feature, they will deploy Message Connector Component. It will add a little complexity for operation and maintenance.
5、Changes
If users want to use this feature, they will deploy Message Connector Component.
6、Architecture
We plan to design the function of Message Connector as a new component. The total architecture is as below:
Enahncement 1: Sub-project: Implement full messages synchronous replication
We will implement the basic feature for this rip. In this phase, Users can use the Message Connector Component for message replication basicly.
Enhancement 2: Sub-project: Monitering some parameters in the process of message replication
This phase we will collect some parameters in the process of message replication, such as message tps, delay, latest synchronous time and so on.User can show this parameters in their own monitering system.
Enhancement 3: Sub-project: Implement exactly once replication
This phase we will implement message idempotency to meet the exactly once demand.
回复: [DISCUSS][RIP-12]Message Connector
Posted by "wangshaojie@cmss.chinamobile.com" <wa...@cmss.chinamobile.com>.
+1
wangshaojie@cmss.chinamobile.com
发件人： huzongtang@cmss.chinamobile.com
发送时间： 2019-03-14 14:31
收件人： users; commits; dev
主题： [DISCUSS][RIP-12]Message Connector
Hi RocketMQ Community,
This is a discuss for a new feature which called Message Connector.The content of this RIP is as following.You can look at it.If you have any question about this rip,you can discuss it with me in the email.
1、Status
Current State: Discuss
Authors: HuZongTang, WangShaoJie
Shepherds: DuHeng
Mailing List discussion: users@rocketmq.apache.org;dev@rocketmq.apache.org
Pull Request:
Released: <5.0.0>
2、Background & Motivation
What do we need to do
In current version of rocketmq project, different cluster can't exchange messages each other. Although dLedger structure improves availability of rocketmq much more, it still can't implement backuping messages for different cluster.
So, we need a mechanism to implement backuping messages for different cluster. It can improve availability of multiple rocketmq cluster. Here, we called it Message Connector.
3、Goals
What problem is this proposal designed to solve?
We plan to design a component call Message Connector for RocketMQ. User can define the topic message routing rule by configing file, such as source topic, targe topic, filter tag , source cluster and targe cluster info.
To what degree should we solve the problem?
(1)Define some rules in configing files;
(2)Better availability for messages in multiple cluster enviroment.
(3)Monitering some parameters in the process of message replication;
4、Non-Goals
What problem is this proposal NOT designed to solve?
In this phase, this rip only support starting message replication from the latest message position currently in the message queue.
Are there any limits to this proposal?
If users want to use this feature, they will deploy Message Connector Component. It will add a little complexity for operation and maintenance.
5、Changes
If users want to use this feature, they will deploy Message Connector Component.
6、Architecture
We plan to design the function of Message Connector as a new component. The total architecture is as below:
Enahncement 1: Sub-project: Implement full messages synchronous replication
We will implement the basic feature for this rip. In this phase, Users can use the Message Connector Component for message replication basicly.
Enhancement 2: Sub-project: Monitering some parameters in the process of message replication
This phase we will collect some parameters in the process of message replication, such as message tps, delay, latest synchronous time and so on.User can show this parameters in their own monitering system.
Enhancement 3: Sub-project: Implement exactly once replication
This phase we will implement message idempotency to meet the exactly once demand.
Re: [DISCUSS][RIP-12]Message Connector
Posted by heng du <du...@gmail.com>.
Message Connector is the long-awaited feature in the community, I am very
glad to see this RIP
huzongtang@cmss.chinamobile.com <hu...@cmss.chinamobile.com>
于2019年3月14日周四 下午2:31写道：
> *Hi RocketMQ Community,*
>
> This is a discuss for a new feature which called Message Connector.The
> content of this RIP is as following.You can look at it.If you have any
> question about this rip,you can discuss it with me in the email.
>
> *1、Status*
>
> - Current State: Discuss
> - Authors: HuZongTang, WangShaoJie
> - Shepherds: DuHeng
> - Mailing List discussion: users@rocketmq.apache.org;
> dev@rocketmq.apache.org
> - Pull Request:
> - Released: <5.0.0>
> -
>
> *2、Background & Motivation*
> *What do we need to do*
>
> In current version of rocketmq project, different cluster can't exchange messages each other. Although dLedger structure improves availability of rocketmq much more, it still can't implement backuping messages for different cluster.
>
> So, we need a mechanism to implement backuping messages for different cluster. It can improve availability of multiple rocketmq cluster. Here, we called it Message Connector.
>
>
> *3、Goals* What problem is this proposal designed to solve?
>
> We plan to design a component call Message Connector for RocketMQ. User can define the topic message routing rule by configing file, such as source topic, targe topic, filter tag , source cluster and targe cluster info.
> To what degree should we solve the problem?
> (1)Define some rules in configing files;
> (2)Better availability for messages in multiple cluster enviroment.
> (3)Monitering some parameters in the process of message replication;
>
>
> *4、Non-Goals* What problem is this proposal NOT designed to solve?
>
> In this phase, this rip only support starting message replication from the latest message position currently in the message queue.
> Are there any limits to this proposal?
>
> If users want to use this feature, they will deploy Message Connector Component. It will add a little complexity for operation and maintenance.
>
>
> *5、Changes*
> If users want to use this feature, they will deploy Message Connector Component.
>
>
> *6、Architecture*
>
> We plan to design the function of Message Connector as a new component. The total architecture is as below:
>
>
>
>
>
>
> *Enahncement 1: Sub-project: Implement full messages synchronous replication*
> We will implement the basic feature for this rip. In this phase, Users can use the Message Connector Component for message replication basicly.
>
>
>
>
> *Enhancement 2: Sub-project: Monitering some parameters in the process of message replication *
> This phase we will collect some parameters in the process of message replication, such as message tps, delay, latest synchronous time and so on.User can show this parameters in their own monitering system.
>
>
>
> *Enhancement 3: Sub-project: Implement exactly once replication *
> This phase we will implement message idempotency to meet the exactly once demand.
>
>
>
回复: [DISCUSS][RIP-12]Message Connector
Posted by "wangshaojie@cmss.chinamobile.com" <wa...@cmss.chinamobile.com>.
+1
wangshaojie@cmss.chinamobile.com
发件人： huzongtang@cmss.chinamobile.com
发送时间： 2019-03-14 14:31
收件人： users; commits; dev
主题： [DISCUSS][RIP-12]Message Connector
Hi RocketMQ Community,
This is a discuss for a new feature which called Message Connector.The content of this RIP is as following.You can look at it.If you have any question about this rip,you can discuss it with me in the email.
1、Status
Current State: Discuss
Authors: HuZongTang, WangShaoJie
Shepherds: DuHeng
Mailing List discussion: users@rocketmq.apache.org;dev@rocketmq.apache.org
Pull Request:
Released: <5.0.0>
2、Background & Motivation
What do we need to do
In current version of rocketmq project, different cluster can't exchange messages each other. Although dLedger structure improves availability of rocketmq much more, it still can't implement backuping messages for different cluster.
So, we need a mechanism to implement backuping messages for different cluster. It can improve availability of multiple rocketmq cluster. Here, we called it Message Connector.
3、Goals
What problem is this proposal designed to solve?
We plan to design a component call Message Connector for RocketMQ. User can define the topic message routing rule by configing file, such as source topic, targe topic, filter tag , source cluster and targe cluster info.
To what degree should we solve the problem?
(1)Define some rules in configing files;
(2)Better availability for messages in multiple cluster enviroment.
(3)Monitering some parameters in the process of message replication;
4、Non-Goals
What problem is this proposal NOT designed to solve?
In this phase, this rip only support starting message replication from the latest message position currently in the message queue.
Are there any limits to this proposal?
If users want to use this feature, they will deploy Message Connector Component. It will add a little complexity for operation and maintenance.
5、Changes
If users want to use this feature, they will deploy Message Connector Component.
6、Architecture
We plan to design the function of Message Connector as a new component. The total architecture is as below:
Enahncement 1: Sub-project: Implement full messages synchronous replication
We will implement the basic feature for this rip. In this phase, Users can use the Message Connector Component for message replication basicly.
Enhancement 2: Sub-project: Monitering some parameters in the process of message replication
This phase we will collect some parameters in the process of message replication, such as message tps, delay, latest synchronous time and so on.User can show this parameters in their own monitering system.
Enhancement 3: Sub-project: Implement exactly once replication
This phase we will implement message idempotency to meet the exactly once demand.
Re: [DISCUSS][RIP-12]Message Connector
Posted by heng du <du...@gmail.com>.
Message Connector is the long-awaited feature in the community, I am very
glad to see this RIP
huzongtang@cmss.chinamobile.com <hu...@cmss.chinamobile.com>
于2019年3月14日周四 下午2:31写道：
> *Hi RocketMQ Community,*
>
> This is a discuss for a new feature which called Message Connector.The
> content of this RIP is as following.You can look at it.If you have any
> question about this rip,you can discuss it with me in the email.
>
> *1、Status*
>
> - Current State: Discuss
> - Authors: HuZongTang, WangShaoJie
> - Shepherds: DuHeng
> - Mailing List discussion: users@rocketmq.apache.org;
> dev@rocketmq.apache.org
> - Pull Request:
> - Released: <5.0.0>
> -
>
> *2、Background & Motivation*
> *What do we need to do*
>
> In current version of rocketmq project, different cluster can't exchange messages each other. Although dLedger structure improves availability of rocketmq much more, it still can't implement backuping messages for different cluster.
>
> So, we need a mechanism to implement backuping messages for different cluster. It can improve availability of multiple rocketmq cluster. Here, we called it Message Connector.
>
>
> *3、Goals* What problem is this proposal designed to solve?
>
> We plan to design a component call Message Connector for RocketMQ. User can define the topic message routing rule by configing file, such as source topic, targe topic, filter tag , source cluster and targe cluster info.
> To what degree should we solve the problem?
> (1)Define some rules in configing files;
> (2)Better availability for messages in multiple cluster enviroment.
> (3)Monitering some parameters in the process of message replication;
>
>
> *4、Non-Goals* What problem is this proposal NOT designed to solve?
>
> In this phase, this rip only support starting message replication from the latest message position currently in the message queue.
> Are there any limits to this proposal?
>
> If users want to use this feature, they will deploy Message Connector Component. It will add a little complexity for operation and maintenance.
>
>
> *5、Changes*
> If users want to use this feature, they will deploy Message Connector Component.
>
>
> *6、Architecture*
>
> We plan to design the function of Message Connector as a new component. The total architecture is as below:
>
>
>
>
>
>
> *Enahncement 1: Sub-project: Implement full messages synchronous replication*
> We will implement the basic feature for this rip. In this phase, Users can use the Message Connector Component for message replication basicly.
>
>
>
>
> *Enhancement 2: Sub-project: Monitering some parameters in the process of message replication *
> This phase we will collect some parameters in the process of message replication, such as message tps, delay, latest synchronous time and so on.User can show this parameters in their own monitering system.
>
>
>
> *Enhancement 3: Sub-project: Implement exactly once replication *
> This phase we will implement message idempotency to meet the exactly once demand.
>
>
>