You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@rocketmq.apache.org by "827404956@qq.com" <82...@qq.com> on 2020/05/25 09:46:01 UTC

RIP-19 Delay Message

RIP-19 RocketMQ DelayMessage

Status

* Current State: Proposed
* Authors:
* Shepherds:
* Mailing List discussion: [RocetMQ Developers](dev@rocketmq.apache.org)
* Pull Request: #PR_NUMBER
* Released: <relased_version>

Background & Motivation

At present, RocketMQ only supports delay messages with fixed delay levels(1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h), which is not flexible enough, and the current delay message design is realized by using new separate Topic(SCHEDULE_TOPIC_XXXX) and Queue, if we want to achieve any time delay of seconds within 40 days, based on the existing solution, 40 * 24 * 60 * 60 = 345600 Queues are needed, and the cost is very high. Also, the message cannot be delayed too long because the CommitLog will be deleted after a certain amount of time (default 3 days)
In order to meet the requirements of flexible time delay messages, this proposal solves this problem by redesigning the architecture of delay messages and introducing TimeWheel.

Goals

Delay message can be delivered in any time delay of seconds within 40 days(or other time range, this should be configured by deployer).

Non-Goals

* Does not support millisecond delay
* Delay message delivery over 40 days will fail
* If the consumer currently has a message accumulation, the delay message will be ranked after the accumulation messages, and will not be delivered strictly according to the configured time

Changes

Architectures

Adopt multi-level TimeWheel design, coarse-grained TimeWheel on disk, fine-grained TimeWheel in RAM. DelayMessageLog stores the delay message sending progress. The normal process is as follows:
1. Message sequence appended to CommitLog
2. The ReputMessageService thread continuously obtains newly added messages, and the Dispatcher determines whether it is a delayed message:
* Normal Message: ConsumeQueue is constructed normally, consumers consume normally
* Delay Message: re-stored to DelayMessageStorage, stored according to the coarser time granularity of the multi-level TimeWheel, such as hour; if the delivery time is in the range of the TimeWheel in the RAM, the message is also added to the TimeWheel in the RAM.
3. At the beginning of the coarse-grained time, the TimeWheel in RAM is constructed according to DelayMessageStorage.
4. Convert the delay message to normal message and append it to CommitLog. If the append is successful, update the delay message delivery record DelayMessageLog.

Implementation Outline

We will implement the proposed changes by 3 phases.

Phase 1

In the master-slave cluster deployment mode, only the master can provide delivery of delay messages

Phase 2

In the master-slave cluster deployment mode, both master and slave can provide delivery of delay messages

Phase 3

In the cluster deployment mode based on Dleger, delay messages can be delivered normally.

HeChengyang

Re: RIP-19 Delay Message

Posted by Rongqian Xu <Ro...@shangri-la.com>.

Dear Developer,

Good work!
Nice feature that RocketMQ missing. We are finally going to put the last picece of the puzzle.

Just add a few comments here:

  *   The arch diagram was broken.
  *   I think the RIP should support delay message within at least 1 year in any seconds.

Thank your contribution on the RocketMQ Project!


From: "827404956@qq.com" <82...@qq.com>
Reply-To: "dev@rocketmq.apache.org" <de...@rocketmq.apache.org>
Date: Monday, May 25, 2020 at 5:51 PM
To: dev <de...@rocketmq.apache.org>, users <us...@rocketmq.apache.org>, commits <co...@rocketmq.apache.org>
Subject: RIP-19 Delay Message

This e-mail is from an external source. Be alert and exercise care.
此邮件发送自公司外部，请保持警惕。


RIP-19 RocketMQ DelayMessage

Status

* Current State: Proposed
* Authors:
* Shepherds:
* Mailing List discussion: [RocetMQ Developers](dev@rocketmq.apache.org)
* Pull Request: #PR_NUMBER
* Released: <relased_version>

Background & Motivation

At present, RocketMQ only supports delay messages with fixed delay levels(1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h), which is not flexible enough, and the current delay message design is realized by using new separate Topic(SCHEDULE_TOPIC_XXXX) and Queue, if we want to achieve any time delay  of seconds within 40 days, based on the existing solution, 40 * 24 * 60 * 60 = 345600 Queues are needed, and the cost is very high. Also, the message cannot be delayed too long because the CommitLog will be deleted after a certain amount of time (default 3 days)
In order to meet the requirements of flexible time delay messages, this proposal solves this problem by redesigning the architecture of delay messages and introducing TimeWheel.

Goals

Delay message can be delivered in any time delay of seconds within 40 days(or other time range, this should be configured by deployer).

Non-Goals

* Does not support millisecond delay
* Delay message delivery over 40 days will fail
* If the consumer currently has a message accumulation, the delay message will be ranked after the accumulation messages, and will not be delivered strictly according to the configured time

Changes

Architectures
[cid:_Foxmail.1@fecfc1d7-137d-ec18-2dbd-81dda82e74b5]
Adopt multi-level TimeWheel design, coarse-grained TimeWheel on disk, fine-grained TimeWheel in RAM. DelayMessageLog stores the delay message sending progress. The normal process is as follows:
1. Message sequence appended to CommitLog
2. The ReputMessageService thread continuously obtains newly added messages, and the Dispatcher determines whether it is a delayed message:
    * Normal Message: ConsumeQueue is constructed normally, consumers consume normally
    * Delay Message: re-stored to DelayMessageStorage, stored according to the coarser time granularity of the multi-level TimeWheel, such as hour; if the delivery time is in the range of the TimeWheel in the RAM, the message is also added to the TimeWheel in the RAM.
3. At the beginning of the coarse-grained time, the TimeWheel in RAM is constructed according to DelayMessageStorage.
4. Convert the delay message to normal message and append it to CommitLog. If the append is successful, update the delay message delivery record DelayMessageLog.

Implementation Outline

We will implement the proposed changes by 3 phases.

Phase 1

In the master-slave cluster deployment mode, only the master can provide delivery of delay messages

Phase 2

In the master-slave cluster deployment mode, both master and slave can provide delivery of delay messages

Phase 3

In the cluster deployment mode based on Dleger, delay messages can be delivered normally.



________________________________
HeChengyang
******************************************************************************
The information contained in this message may be privileged and confidential and protected from disclosure. Any dissemination, distribution, copying of or taking of any action in reliance upon this message, other than to/by the addressee, is strictly prohibited.
If you have received this in error, please destroy it and notify us immediately. Opinions, conclusions and other information in this message that do not relate to the official business of our company shall be understood as neither given nor endorsed by it and we disclaim all liability for any such statements.
******************************************************************************