You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by ma...@gmail.com on 2023/04/18 02:07:55 UTC
[DISCUSS] Sorting out pulsar's internal thread pools
Hello, folks.
I would like to start discussing the pulsar internal thread pool sorting out.
How did I get this idea?
Recently, we met some problems with the BK operation timeout. After investigating, we found an issue that is we share the IO executor(workgroup) with the Bookkeeper client and internal client and do some other async task in the dispatcher or somewhere to avoid deadlock.
But the problem over here. If we use this executor to do some kind of `blocking`(or spend much time computing. e.g. reply to many delayed messages) operation, it will block BK clients from sending requests if they are using the same thread.
And then, I checked all the usage of the thread pool. We need the rule to constrain what thread pool we should use.
What am I expecting?
I want to collect all the thread pools and define a clear usage guide to avoid wrong use and improve the fault tolerance(the component problem shouldn't affect the whole broker)
I need to hear your guy's opinions. Please feel free to leave any questions. Thanks!
Best,
Mattison
Re: [DISCUSS] Sorting out pulsar's internal thread pools
Posted by Zike Yang <zi...@apache.org>.
Sounds great. +1
Thanks,
Zike Yang
On Tue, Apr 18, 2023 at 2:37 PM Xiangying Meng <xi...@apache.org> wrote:
>
> Thank you for bringing up this important topic. I completely agree with
> this initiative.
> This would be a great starting point for revisiting and improving the
> Pulsar codebase.
>
> Thanks,
> Xiangying
>
> On Tue, Apr 18, 2023 at 2:18 PM Lin Lin <li...@apache.org> wrote:
>
> > This is a good idea.
> >
> > Thanks,
> > Lin Lin
> >
> > On 2023/04/18 02:07:55 mattisonchao@gmail.com wrote:
> > >
> > > Hello, folks.
> > >
> > > I would like to start discussing the pulsar internal thread pool sorting
> > out.
> > >
> > > How did I get this idea?
> > >
> > > Recently, we met some problems with the BK operation timeout. After
> > investigating, we found an issue that is we share the IO
> > executor(workgroup) with the Bookkeeper client and internal client and do
> > some other async task in the dispatcher or somewhere to avoid deadlock.
> > >
> > > But the problem over here. If we use this executor to do some kind of
> > `blocking`(or spend much time computing. e.g. reply to many delayed
> > messages) operation, it will block BK clients from sending requests if they
> > are using the same thread.
> > >
> > > And then, I checked all the usage of the thread pool. We need the rule
> > to constrain what thread pool we should use.
> > >
> > > What am I expecting?
> > >
> > > I want to collect all the thread pools and define a clear usage guide to
> > avoid wrong use and improve the fault tolerance(the component problem
> > shouldn't affect the whole broker)
> > >
> > >
> > >
> > > I need to hear your guy's opinions. Please feel free to leave any
> > questions. Thanks!
> > >
> > >
> > > Best,
> > > Mattison
> > >
> > >
> > >
> >
Re: [DISCUSS] Sorting out pulsar's internal thread pools
Posted by Xiangying Meng <xi...@apache.org>.
Thank you for bringing up this important topic. I completely agree with
this initiative.
This would be a great starting point for revisiting and improving the
Pulsar codebase.
Thanks,
Xiangying
On Tue, Apr 18, 2023 at 2:18 PM Lin Lin <li...@apache.org> wrote:
> This is a good idea.
>
> Thanks,
> Lin Lin
>
> On 2023/04/18 02:07:55 mattisonchao@gmail.com wrote:
> >
> > Hello, folks.
> >
> > I would like to start discussing the pulsar internal thread pool sorting
> out.
> >
> > How did I get this idea?
> >
> > Recently, we met some problems with the BK operation timeout. After
> investigating, we found an issue that is we share the IO
> executor(workgroup) with the Bookkeeper client and internal client and do
> some other async task in the dispatcher or somewhere to avoid deadlock.
> >
> > But the problem over here. If we use this executor to do some kind of
> `blocking`(or spend much time computing. e.g. reply to many delayed
> messages) operation, it will block BK clients from sending requests if they
> are using the same thread.
> >
> > And then, I checked all the usage of the thread pool. We need the rule
> to constrain what thread pool we should use.
> >
> > What am I expecting?
> >
> > I want to collect all the thread pools and define a clear usage guide to
> avoid wrong use and improve the fault tolerance(the component problem
> shouldn't affect the whole broker)
> >
> >
> >
> > I need to hear your guy's opinions. Please feel free to leave any
> questions. Thanks!
> >
> >
> > Best,
> > Mattison
> >
> >
> >
>
Re: [DISCUSS] Sorting out pulsar's internal thread pools
Posted by Lin Lin <li...@apache.org>.
This is a good idea.
Thanks,
Lin Lin
On 2023/04/18 02:07:55 mattisonchao@gmail.com wrote:
>
> Hello, folks.
>
> I would like to start discussing the pulsar internal thread pool sorting out.
>
> How did I get this idea?
>
> Recently, we met some problems with the BK operation timeout. After investigating, we found an issue that is we share the IO executor(workgroup) with the Bookkeeper client and internal client and do some other async task in the dispatcher or somewhere to avoid deadlock.
>
> But the problem over here. If we use this executor to do some kind of `blocking`(or spend much time computing. e.g. reply to many delayed messages) operation, it will block BK clients from sending requests if they are using the same thread.
>
> And then, I checked all the usage of the thread pool. We need the rule to constrain what thread pool we should use.
>
> What am I expecting?
>
> I want to collect all the thread pools and define a clear usage guide to avoid wrong use and improve the fault tolerance(the component problem shouldn't affect the whole broker)
>
>
>
> I need to hear your guy's opinions. Please feel free to leave any questions. Thanks!
>
>
> Best,
> Mattison
>
>
>
Re: [DISCUSS] Sorting out pulsar's internal thread pools
Posted by Asaf Mesika <as...@gmail.com>.
I would check with the veterans like Matteo to see what was the initial
reasoning behind sharing threads with the BK client.
On Tue, Apr 18, 2023 at 1:59 PM Cong Zhao <zh...@apache.org> wrote:
> I think this is a good idea for the reasonable use of pulsar threads.
>
> Thanks,
> Cong Zhao
>
> On 2023/04/18 02:07:55 mattisonchao@gmail.com wrote:
> >
> > Hello, folks.
> >
> > I would like to start discussing the pulsar internal thread pool sorting
> out.
> >
> > How did I get this idea?
> >
> > Recently, we met some problems with the BK operation timeout. After
> investigating, we found an issue that is we share the IO
> executor(workgroup) with the Bookkeeper client and internal client and do
> some other async task in the dispatcher or somewhere to avoid deadlock.
> >
> > But the problem over here. If we use this executor to do some kind of
> `blocking`(or spend much time computing. e.g. reply to many delayed
> messages) operation, it will block BK clients from sending requests if they
> are using the same thread.
> >
> > And then, I checked all the usage of the thread pool. We need the rule
> to constrain what thread pool we should use.
> >
> > What am I expecting?
> >
> > I want to collect all the thread pools and define a clear usage guide to
> avoid wrong use and improve the fault tolerance(the component problem
> shouldn't affect the whole broker)
> >
> >
> >
> > I need to hear your guy's opinions. Please feel free to leave any
> questions. Thanks!
> >
> >
> > Best,
> > Mattison
> >
> >
> >
>
Re: [DISCUSS] Sorting out pulsar's internal thread pools
Posted by Cong Zhao <zh...@apache.org>.
I think this is a good idea for the reasonable use of pulsar threads.
Thanks,
Cong Zhao
On 2023/04/18 02:07:55 mattisonchao@gmail.com wrote:
>
> Hello, folks.
>
> I would like to start discussing the pulsar internal thread pool sorting out.
>
> How did I get this idea?
>
> Recently, we met some problems with the BK operation timeout. After investigating, we found an issue that is we share the IO executor(workgroup) with the Bookkeeper client and internal client and do some other async task in the dispatcher or somewhere to avoid deadlock.
>
> But the problem over here. If we use this executor to do some kind of `blocking`(or spend much time computing. e.g. reply to many delayed messages) operation, it will block BK clients from sending requests if they are using the same thread.
>
> And then, I checked all the usage of the thread pool. We need the rule to constrain what thread pool we should use.
>
> What am I expecting?
>
> I want to collect all the thread pools and define a clear usage guide to avoid wrong use and improve the fault tolerance(the component problem shouldn't affect the whole broker)
>
>
>
> I need to hear your guy's opinions. Please feel free to leave any questions. Thanks!
>
>
> Best,
> Mattison
>
>
>
Re: [DISCUSS] Sorting out pulsar's internal thread pools
Posted by Yunze Xu <yz...@streamnative.io.INVALID>.
I agree with this improvement. I had the same idea before but it would
take too much time so I didn't continue at that time.
Thanks,
Yunze
On Tue, Apr 18, 2023 at 10:08 AM <ma...@gmail.com> wrote:
>
>
> Hello, folks.
>
> I would like to start discussing the pulsar internal thread pool sorting out.
>
> How did I get this idea?
>
> Recently, we met some problems with the BK operation timeout. After investigating, we found an issue that is we share the IO executor(workgroup) with the Bookkeeper client and internal client and do some other async task in the dispatcher or somewhere to avoid deadlock.
>
> But the problem over here. If we use this executor to do some kind of `blocking`(or spend much time computing. e.g. reply to many delayed messages) operation, it will block BK clients from sending requests if they are using the same thread.
>
> And then, I checked all the usage of the thread pool. We need the rule to constrain what thread pool we should use.
>
> What am I expecting?
>
> I want to collect all the thread pools and define a clear usage guide to avoid wrong use and improve the fault tolerance(the component problem shouldn't affect the whole broker)
>
>
>
> I need to hear your guy's opinions. Please feel free to leave any questions. Thanks!
>
>
> Best,
> Mattison
>
>