You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by 2017赵科 <46...@qq.com.INVALID> on 2023/02/21 06:37:38 UTC
回复： [DISCUSS][RIP-62] Cold read control
Dear predecessors
Thank you very much for your reply at first, the reason that add the logical cold data judgment is we can use this to distinguish logic hot data and logic cold data.
The logic hot data that is commonly Newly produced and a lot of normal consumers will read it, Although this part of data are used to stay in the pagecache(physical hot data).
So we decide that when consumers read this part of data, can not be in cold data read control, let them read quickly. The same can be obtained，the logic cold data that is commonly old produced and a little of unnormal consumers will read it and this part of data are not in the pagecache. if some unnormal consumers read logic cold data when they are not controlled, the system disk IO stress will be high, it will influence the whole broker's normaly reading and writing.
Your response would be highly appreciated.
Thanks,
drizzle.zk
2017赵科
464473306@qq.com
------------------ 原始邮件 ------------------
发件人: "dev" <dongyuanpan0@gmail.com>;
发送时间: 2023年2月17日(星期五) 下午3:56
收件人: "dev"<dev@rocketmq.apache.org>;
主题: Re: [DISCUSS][RIP-62] Cold read control
That's a good idea. But I don't understand why add the logical cold
data judgment? Physical cold data reading will have a practical
impact, but logical cold data may only be physical cold data, not
necessarily cold data, and its reading does not necessarily cause
system stress. One point that can be thought of is to speed up cold
data judgment?
SSpirits <admin@lv5.moe> 于2023年2月14日周二 11:54写道：
>
> Hi drizzle.zk,
>
> Currently, we have a precise way to find out cold data, please see https://github.com/apache/rocketmq/issues/5837. And I agree with you that we should stop prefetch CommitLog if it is not the latest file. Maybe we can use madvise system call and MADV_FREE or MADV_DONTNEED flag.
>
> Your response would be highly appreciated.
> SSpirits
> Email：admin@lv5.moe<mailto:admin@lv5.moe>
> 在 2023年2月10日 +0800 18:00，2017赵科 <464473306@qq.com.INVALID>，写道：
> Hello RocketMQ Community:
>
>
> We have plans to add Cold read control features can control the reading speed of cold data so as to ensure the stability of the system.
> We also have written the proposal, and you can see it by the link below:
> https://docs.google.com/document/d/1P8JPX5b9CwYMlaFi_UBxqwxFoIiOXWuAEjV4yWdfFUw/edit?usp=sharing
>
>
> Please welcome to reply to this email or comment on the proposal if you have any questions or suggestions.
>
>
>
> Thanks,
> drizzle.zk
>
>
>
>
>
>
>
> 2017赵科
> 464473306@qq.com
>
>
>
> &nbsp;
Re: [DISCUSS][RIP-62] Cold read control
Posted by fuyou <fu...@gmail.com>.
+1
2017赵科 <46...@qq.com.invalid> 于2023年2月21日周二 14:37写道：
> Dear predecessors
>
>
> Thank you very much for your reply at first, the reason that add the
> logical cold data judgment is we can use this to distinguish logic hot data
> and logic cold data.
> The logic hot data that is commonly Newly produced and a lot of normal
> consumers will read it, Although this part of data are used to stay in the
> pagecache(physical hot data).
> So we decide that when consumers read this part of data, can not be in
> cold data read control, let them read quickly. The same can be obtained，the
> logic cold data that is commonly old produced and a little of unnormal
> consumers will read it and this part of data are not in the pagecache. if
> some unnormal consumers read logic cold data when they are not controlled,
> the system disk IO stress will be high, it will influence the whole
> broker's normaly reading and writing.
>
>
> Your response would be highly appreciated.
>
>
> Thanks,
> drizzle.zk
>
>
> 2017赵科
> 464473306@qq.com
>
>
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人:
> "dev"
> <
> dongyuanpan0@gmail.com>;
> 发送时间: 2023年2月17日(星期五) 下午3:56
> 收件人: "dev"<dev@rocketmq.apache.org>;
>
> 主题: Re: [DISCUSS][RIP-62] Cold read control
>
>
>
> That's a good idea. But I don't understand why add the logical cold
> data judgment? Physical cold data reading will have a practical
> impact, but logical cold data may only be physical cold data, not
> necessarily cold data, and its reading does not necessarily cause
> system stress. One point that can be thought of is to speed up cold
> data judgment?
>
> SSpirits <admin@lv5.moe> 于2023年2月14日周二 11:54写道：
> >
> > Hi drizzle.zk,
> >
> > Currently, we have a precise way to find out cold data, please see
> https://github.com/apache/rocketmq/issues/5837. And I agree with you that
> we should stop prefetch CommitLog if it is not the latest file. Maybe we
> can use madvise system call and MADV_FREE or MADV_DONTNEED flag.
> >
> > Your response would be highly appreciated.
> > SSpirits
> > Email：admin@lv5.moe<mailto:admin@lv5.moe>
> > 在 2023年2月10日 +0800 18:00，2017赵科 <464473306@qq.com.INVALID>，写道：
> > Hello RocketMQ Community:
> >
> >
> > We have plans to add Cold read control features can control the
> reading speed of cold data so as to ensure the stability of the system.
> > We also have written the proposal, and you can see it by the link
> below:
> >
> https://docs.google.com/document/d/1P8JPX5b9CwYMlaFi_UBxqwxFoIiOXWuAEjV4yWdfFUw/edit?usp=sharing
> >
> <https://docs.google.com/document/d/1P8JPX5b9CwYMlaFi_UBxqwxFoIiOXWuAEjV4yWdfFUw/edit?usp=sharing>>
> ;
> >
> > Please welcome to reply to this email or comment on the proposal if
> you have any questions or suggestions.
> >
> >
> >
> > Thanks,
> > drizzle.zk
> >
> >
> >
> >
> >
> >
> >
> > 2017赵科
> > 464473306@qq.com
> >
> >
> >
> > &nbsp;
--
=============================================
fuyou001
Best Regards