You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by "nowinkey@tom.com" <no...@tom.com> on 2022/10/17 06:54:48 UTC
[DISCUSS] [RIP-52 Optimize Building ConsumeQueue]
Hi, RocketMQ Community
The generation of messages depend on the ReputMessageService single thread. When the message production traffic is heavy, the ConsumeQueue messages are generated slowly, resulting in a delay in message consumption.
We need to process CommitLog messages concurrently to speed up the generation of ConsumeQueue messages.
[RIP-52 Optimize Building ConsumeQueue]
Wiki <https://github.com/apache/rocketmq/wiki/RIP-52-Optimize-Building-ConsumeQueue>
Chinese version <https://yu7y22ce7k.feishu.cn/docx/doxcnltrB7VzUKCqx0yxqMgJ4RM>
English version <https://yu7y22ce7k.feishu.cn/docx/doxcn9A5lPA05AD0oISoB11FL2f>
Please reply to this email if you have any questions
Best Regards
nowinkey@tom.com
Re: Re: [DISCUSS] [RIP-52 Optimize Building ConsumeQueue]
Posted by "nowinkey@tom.com" <no...@tom.com>.
Hi, lizhanhui
Nice idea. This way we can collect the time between CommitLog and ConsumeQueue for a single message. Then we can know how late the message is.
Now do we need to write the test method and write the data to get that data?
nowinkey@tom.com
> @nowinkey@tom.com <no...@tom.com> Can we, preliminarily, decompose the
> dispatch request time into multiple slices, like Linux Kernel IO
> stages(await time, service time...)? This way, guys would roughly get goals
> of the optimization.
> For example, prepare a handful of commit log files without associated
> consume queues, run the recovery process. During the recovery, collect
> statistics of each DispatchRequest, including duration in queue, time for
> writing consume queue logs
Re: [DISCUSS] [RIP-52 Optimize Building ConsumeQueue]
Posted by Zhanhui Li <li...@gmail.com>.
Hi aaron.ai,
What kind of specific performance profiling data do you expect? Basically,
this RIP is changing single thread ReputMessageService to
multi-stage-multi-threaded version. Something similar in the Linux world is
multiple-queuing block IO mechanism
https://docs.kernel.org/block/blk-mq.html
@nowinkey@tom.com <no...@tom.com> Can we, preliminarily, decompose the
dispatch request time into multiple slices, like Linux Kernel IO
stages(await time, service time...)? This way, guys would roughly get goals
of the optimization.
For example, prepare a handful of commit log files without associated
consume queues, run the recovery process. During the recovery, collect
statistics of each DispatchRequest, including duration in queue, time for
writing consume queue logs
Zhanhui Li
On Mon, Oct 17, 2022 at 5:03 PM aaron ai <ya...@gmail.com> wrote:
> Hi, Could you provide more data(performance profiling or another way) to
> support this proposal ?
>
> fuyou <fu...@gmail.com> 于2022年10月17日周一 15:33写道：
>
> > good idea.
> >
> > but i think Introducing bitmap is complicated, so what's about using hash
> > for topic and share some queue,ReputService access queue to build Consume
> > Queue.
> >
> > Zhanhui Li <li...@gmail.com> 于2022年10月17日周一 15:04写道：
> >
> > > Hey, RocketMQ developers and users,
> > >
> > > It's glad and encouraging to see improvement in the building of
> > > ConsumeQueue, which is essential to keep end-to-end latency low, even
> if
> > > workload of RocketMQ broker is pretty heavy. This is one of the places
> > > where RocketMQ really shines among other messaging/streaming products.
> > >
> > > Questions, suggestions and comments are all welcome!
> > >
> > > Best Regards!
> > >
> > > Zhanhui Li
> > >
> > > On Mon, Oct 17, 2022 at 2:55 PM nowinkey@tom.com <no...@tom.com>
> > wrote:
> > >
> > > > Hi, RocketMQ Community
> > > >
> > > > The generation of messages depend on the ReputMessageService single
> > > > thread. When the message production traffic is heavy, the
> ConsumeQueue
> > > > messages are generated slowly, resulting in a delay in message
> > > consumption.
> > > >
> > > > We need to process CommitLog messages concurrently to speed up the
> > > > generation of ConsumeQueue messages.
> > > >
> > > > [RIP-52 Optimize Building ConsumeQueue]
> > > > Wiki <
> > > >
> > >
> >
> https://github.com/apache/rocketmq/wiki/RIP-52-Optimize-Building-ConsumeQueue
> > > > >
> > > > Chinese version <
> > > > https://yu7y22ce7k.feishu.cn/docx/doxcnltrB7VzUKCqx0yxqMgJ4RM>
> > > > English version <
> > > > https://yu7y22ce7k.feishu.cn/docx/doxcn9A5lPA05AD0oISoB11FL2f>
> > > >
> > > > Please reply to this email if you have any questions
> > > > Best Regards
> > > >
> > > >
> > > > nowinkey@tom.com
> > > >
> > >
> >
> >
> > --
> > =============================================
> >
> > fuyou001
> > Best Regards
> >
>
Re: [DISCUSS] [RIP-52 Optimize Building ConsumeQueue]
Posted by aaron ai <ya...@gmail.com>.
Hi, Could you provide more data(performance profiling or another way) to
support this proposal ?
fuyou <fu...@gmail.com> 于2022年10月17日周一 15:33写道：
> good idea.
>
> but i think Introducing bitmap is complicated, so what's about using hash
> for topic and share some queue,ReputService access queue to build Consume
> Queue.
>
> Zhanhui Li <li...@gmail.com> 于2022年10月17日周一 15:04写道：
>
> > Hey, RocketMQ developers and users,
> >
> > It's glad and encouraging to see improvement in the building of
> > ConsumeQueue, which is essential to keep end-to-end latency low, even if
> > workload of RocketMQ broker is pretty heavy. This is one of the places
> > where RocketMQ really shines among other messaging/streaming products.
> >
> > Questions, suggestions and comments are all welcome!
> >
> > Best Regards!
> >
> > Zhanhui Li
> >
> > On Mon, Oct 17, 2022 at 2:55 PM nowinkey@tom.com <no...@tom.com>
> wrote:
> >
> > > Hi, RocketMQ Community
> > >
> > > The generation of messages depend on the ReputMessageService single
> > > thread. When the message production traffic is heavy, the ConsumeQueue
> > > messages are generated slowly, resulting in a delay in message
> > consumption.
> > >
> > > We need to process CommitLog messages concurrently to speed up the
> > > generation of ConsumeQueue messages.
> > >
> > > [RIP-52 Optimize Building ConsumeQueue]
> > > Wiki <
> > >
> >
> https://github.com/apache/rocketmq/wiki/RIP-52-Optimize-Building-ConsumeQueue
> > > >
> > > Chinese version <
> > > https://yu7y22ce7k.feishu.cn/docx/doxcnltrB7VzUKCqx0yxqMgJ4RM>
> > > English version <
> > > https://yu7y22ce7k.feishu.cn/docx/doxcn9A5lPA05AD0oISoB11FL2f>
> > >
> > > Please reply to this email if you have any questions
> > > Best Regards
> > >
> > >
> > > nowinkey@tom.com
> > >
> >
>
>
> --
> =============================================
>
> fuyou001
> Best Regards
>
Re: Re: [DISCUSS] [RIP-52 Optimize Building ConsumeQueue]
Posted by "nowinkey@tom.com" <no...@tom.com>.
Hi, fuyou,
My idea is to use Hashmap<topic, Hashmap<queueId, bitmap>> map. I think bitmap is not complicated and save memory.
I don't quite understand your specific idea, is it the same as mine.
nowinkey@tom.com
From: fuyou
Date: 2022-10-17 15:33
To: dev
Subject: Re: [DISCUSS] [RIP-52 Optimize Building ConsumeQueue]
good idea.
but i think Introducing bitmap is complicated, so what's about using hash
for topic and share some queue,ReputService access queue to build Consume
Queue.
Zhanhui Li <li...@gmail.com> 于2022年10月17日周一 15:04写道：
> Hey, RocketMQ developers and users,
>
> It's glad and encouraging to see improvement in the building of
> ConsumeQueue, which is essential to keep end-to-end latency low, even if
> workload of RocketMQ broker is pretty heavy. This is one of the places
> where RocketMQ really shines among other messaging/streaming products.
>
> Questions, suggestions and comments are all welcome!
>
> Best Regards!
>
> Zhanhui Li
>
> On Mon, Oct 17, 2022 at 2:55 PM nowinkey@tom.com <no...@tom.com> wrote:
>
> > Hi, RocketMQ Community
> >
> > The generation of messages depend on the ReputMessageService single
> > thread. When the message production traffic is heavy, the ConsumeQueue
> > messages are generated slowly, resulting in a delay in message
> consumption.
> >
> > We need to process CommitLog messages concurrently to speed up the
> > generation of ConsumeQueue messages.
> >
> > [RIP-52 Optimize Building ConsumeQueue]
> > Wiki <
> >
> https://github.com/apache/rocketmq/wiki/RIP-52-Optimize-Building-ConsumeQueue
> > >
> > Chinese version <
> > https://yu7y22ce7k.feishu.cn/docx/doxcnltrB7VzUKCqx0yxqMgJ4RM>
> > English version <
> > https://yu7y22ce7k.feishu.cn/docx/doxcn9A5lPA05AD0oISoB11FL2f>
> >
> > Please reply to this email if you have any questions
> > Best Regards
> >
> >
> > nowinkey@tom.com
> >
>
--
=============================================
fuyou001
Best Regards
Re: [DISCUSS] [RIP-52 Optimize Building ConsumeQueue]
Posted by fuyou <fu...@gmail.com>.
good idea.
but i think Introducing bitmap is complicated, so what's about using hash
for topic and share some queue,ReputService access queue to build Consume
Queue.
Zhanhui Li <li...@gmail.com> 于2022年10月17日周一 15:04写道：
> Hey, RocketMQ developers and users,
>
> It's glad and encouraging to see improvement in the building of
> ConsumeQueue, which is essential to keep end-to-end latency low, even if
> workload of RocketMQ broker is pretty heavy. This is one of the places
> where RocketMQ really shines among other messaging/streaming products.
>
> Questions, suggestions and comments are all welcome!
>
> Best Regards!
>
> Zhanhui Li
>
> On Mon, Oct 17, 2022 at 2:55 PM nowinkey@tom.com <no...@tom.com> wrote:
>
> > Hi, RocketMQ Community
> >
> > The generation of messages depend on the ReputMessageService single
> > thread. When the message production traffic is heavy, the ConsumeQueue
> > messages are generated slowly, resulting in a delay in message
> consumption.
> >
> > We need to process CommitLog messages concurrently to speed up the
> > generation of ConsumeQueue messages.
> >
> > [RIP-52 Optimize Building ConsumeQueue]
> > Wiki <
> >
> https://github.com/apache/rocketmq/wiki/RIP-52-Optimize-Building-ConsumeQueue
> > >
> > Chinese version <
> > https://yu7y22ce7k.feishu.cn/docx/doxcnltrB7VzUKCqx0yxqMgJ4RM>
> > English version <
> > https://yu7y22ce7k.feishu.cn/docx/doxcn9A5lPA05AD0oISoB11FL2f>
> >
> > Please reply to this email if you have any questions
> > Best Regards
> >
> >
> > nowinkey@tom.com
> >
>
--
=============================================
fuyou001
Best Regards
Re: [DISCUSS] [RIP-52 Optimize Building ConsumeQueue]
Posted by Zhanhui Li <li...@gmail.com>.
Hey, RocketMQ developers and users,
It's glad and encouraging to see improvement in the building of
ConsumeQueue, which is essential to keep end-to-end latency low, even if
workload of RocketMQ broker is pretty heavy. This is one of the places
where RocketMQ really shines among other messaging/streaming products.
Questions, suggestions and comments are all welcome!
Best Regards!
Zhanhui Li
On Mon, Oct 17, 2022 at 2:55 PM nowinkey@tom.com <no...@tom.com> wrote:
> Hi, RocketMQ Community
>
> The generation of messages depend on the ReputMessageService single
> thread. When the message production traffic is heavy, the ConsumeQueue
> messages are generated slowly, resulting in a delay in message consumption.
>
> We need to process CommitLog messages concurrently to speed up the
> generation of ConsumeQueue messages.
>
> [RIP-52 Optimize Building ConsumeQueue]
> Wiki <
> https://github.com/apache/rocketmq/wiki/RIP-52-Optimize-Building-ConsumeQueue
> >
> Chinese version <
> https://yu7y22ce7k.feishu.cn/docx/doxcnltrB7VzUKCqx0yxqMgJ4RM>
> English version <
> https://yu7y22ce7k.feishu.cn/docx/doxcn9A5lPA05AD0oISoB11FL2f>
>
> Please reply to this email if you have any questions
> Best Regards
>
>
> nowinkey@tom.com
>