[jira] [Updated] (ROCKETMQ-67) Consistent Hash allocate strategy support

["Jaskey Lam (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ROCKETMQ-67?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jaskey Lam updated ROCKETMQ-67:
-------------------------------
    Description: 
For now, the average allocate strategy is very sensitive when clients register and unrigister.

A Consistent Hash allocate strategy option is valueable for the developers who care more about latency stabilization and messages duplication.


Intentions: 
The default AllocateMessageQueueStrategy is averaging strategy which allocate queue to consumer as evenly as possible. Whenever queues numbers or consumer numbers changed, say a new consumer starts or an old consumer shutdowns, a rehashing will be triggered then almost all consumer suffered from this that they will rebalance to drop old queues and get new queues.

And that will cause

message latency from producer to consumer increases at the moment when consumer/queue numbers change, even when they scale up.

messages will be duplicated significantly since the offset may not be persisted to broker and that queue is assigned to another consumer to pull messages from.

This is especially significant when they have tens of consumer instances and scale-up or deployment is often.

Consistent Hash strategy to allocate queue is a good choice for these users.

  was:
For now, the average allocate strategy is very sensitive when clients register and unrigister.

A Consistent Hash allocate strategy is valueable for the developer who cares more about  stabilization than averaging.

 This option should be supported. And averaging should be meet as much as possible.


> Consistent Hash allocate strategy support
> -----------------------------------------
>
>                 Key: ROCKETMQ-67
>                 URL: https://issues.apache.org/jira/browse/ROCKETMQ-67
>             Project: Apache RocketMQ
>          Issue Type: New Feature
>          Components: rocketmq-client
>            Reporter: Jaskey Lam
>            Assignee: Jaskey Lam
>             Fix For: 4.1.0-incubating
>
>
> For now, the average allocate strategy is very sensitive when clients register and unrigister.
> A Consistent Hash allocate strategy option is valueable for the developers who care more about latency stabilization and messages duplication.
> Intentions: 
> The default AllocateMessageQueueStrategy is averaging strategy which allocate queue to consumer as evenly as possible. Whenever queues numbers or consumer numbers changed, say a new consumer starts or an old consumer shutdowns, a rehashing will be triggered then almost all consumer suffered from this that they will rebalance to drop old queues and get new queues.
> And that will cause
> message latency from producer to consumer increases at the moment when consumer/queue numbers change, even when they scale up.
> messages will be duplicated significantly since the offset may not be persisted to broker and that queue is assigned to another consumer to pull messages from.
> This is especially significant when they have tens of consumer instances and scale-up or deployment is often.
> Consistent Hash strategy to allocate queue is a good choice for these users.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)