You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by 丛搏 <bo...@apache.org> on 2022/10/25 04:24:39 UTC
[DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Hi, pulsar community:
I start a PIP about `User-friendly acknowledgeCumulative API on a
partitioned topic or multi-topics`
discuss thread: https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
PIP: https://github.com/apache/pulsar/issues/18182
Thanks,
bo
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by 丛搏 <co...@gmail.com>.
In the future, I will add a config for `BatchReceivePolicy.class`, I
haven't thought of a name yet. The config will control the
`MultiTopicsConsumerImpl.class` whether use
`ConsumerImpl.class`.batchReceive() directly or use
`ConsumerBase.class` `incomingMessages`.
Asaf Mesika <as...@gmail.com> 于2022年11月8日周二 21:30写道：
>
> Ok.
>
> I would list what you wrote in your original PIP for context.
>
> What is your future improvement you plan?
>
> On Thu, Nov 3, 2022 at 11:46 AM 丛搏 <co...@gmail.com> wrote:
>
> > Good question!
> > It is a scene for our subsequent optimization, and we are implementing
> > it with minimal changes.
> >
> > Asaf Mesika <as...@gmail.com> 于2022年11月3日周四 16:49写道：
> > >
> > > Regarding the implementation suggested in the PIP:
> > >
> > > It said that you will read the messages from all partitions assigned, and
> > > once you see a message from a different partition you halt and supply the
> > > batch.
> > > My question is: Why can't we simply read the messages from one partition
> > > (say 10 messages) and then the next read from the other partition?
> > The amount of data in a partition is not large, and there are many
> > partitions. In this scenario, this method may have a large delay.
> > >
> > > On Thu, Oct 27, 2022 at 3:36 PM Haiting Jiang <ji...@gmail.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Haiting
> > > >
> > > > On Thu, Oct 27, 2022 at 5:55 PM Yubiao Feng
> > > > <yu...@streamnative.io.invalid> wrote:
> > > > >
> > > > > +1
> > > > >
> > > > > On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
> > > > >
> > > > > > Hi, pulsar community:
> > > > > >
> > > > > > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > > > > > partitioned topic or multi-topics`
> > > > > >
> > > > > > discuss thread:
> > > > > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > > > > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > > > >
> > > > > > PIP: https://github.com/apache/pulsar/issues/18182
> > > > > >
> > > > > > Thanks,
> > > > > > bo
> > > > > >
> > > >
> >
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by Asaf Mesika <as...@gmail.com>.
Ok.
I would list what you wrote in your original PIP for context.
What is your future improvement you plan?
On Thu, Nov 3, 2022 at 11:46 AM 丛搏 <co...@gmail.com> wrote:
> Good question!
> It is a scene for our subsequent optimization, and we are implementing
> it with minimal changes.
>
> Asaf Mesika <as...@gmail.com> 于2022年11月3日周四 16:49写道：
> >
> > Regarding the implementation suggested in the PIP:
> >
> > It said that you will read the messages from all partitions assigned, and
> > once you see a message from a different partition you halt and supply the
> > batch.
> > My question is: Why can't we simply read the messages from one partition
> > (say 10 messages) and then the next read from the other partition?
> The amount of data in a partition is not large, and there are many
> partitions. In this scenario, this method may have a large delay.
> >
> > On Thu, Oct 27, 2022 at 3:36 PM Haiting Jiang <ji...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > Haiting
> > >
> > > On Thu, Oct 27, 2022 at 5:55 PM Yubiao Feng
> > > <yu...@streamnative.io.invalid> wrote:
> > > >
> > > > +1
> > > >
> > > > On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
> > > >
> > > > > Hi, pulsar community:
> > > > >
> > > > > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > > > > partitioned topic or multi-topics`
> > > > >
> > > > > discuss thread:
> > > > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > > > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > > >
> > > > > PIP: https://github.com/apache/pulsar/issues/18182
> > > > >
> > > > > Thanks,
> > > > > bo
> > > > >
> > >
>
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by 丛搏 <co...@gmail.com>.
Good question!
It is a scene for our subsequent optimization, and we are implementing
it with minimal changes.
Asaf Mesika <as...@gmail.com> 于2022年11月3日周四 16:49写道：
>
> Regarding the implementation suggested in the PIP:
>
> It said that you will read the messages from all partitions assigned, and
> once you see a message from a different partition you halt and supply the
> batch.
> My question is: Why can't we simply read the messages from one partition
> (say 10 messages) and then the next read from the other partition?
The amount of data in a partition is not large, and there are many
partitions. In this scenario, this method may have a large delay.
>
> On Thu, Oct 27, 2022 at 3:36 PM Haiting Jiang <ji...@gmail.com>
> wrote:
>
> > +1
> >
> > Haiting
> >
> > On Thu, Oct 27, 2022 at 5:55 PM Yubiao Feng
> > <yu...@streamnative.io.invalid> wrote:
> > >
> > > +1
> > >
> > > On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
> > >
> > > > Hi, pulsar community:
> > > >
> > > > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > > > partitioned topic or multi-topics`
> > > >
> > > > discuss thread:
> > > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > >
> > > > PIP: https://github.com/apache/pulsar/issues/18182
> > > >
> > > > Thanks,
> > > > bo
> > > >
> >
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by Asaf Mesika <as...@gmail.com>.
Regarding the implementation suggested in the PIP:
It said that you will read the messages from all partitions assigned, and
once you see a message from a different partition you halt and supply the
batch.
My question is: Why can't we simply read the messages from one partition
(say 10 messages) and then the next read from the other partition?
On Thu, Oct 27, 2022 at 3:36 PM Haiting Jiang <ji...@gmail.com>
wrote:
> +1
>
> Haiting
>
> On Thu, Oct 27, 2022 at 5:55 PM Yubiao Feng
> <yu...@streamnative.io.invalid> wrote:
> >
> > +1
> >
> > On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
> >
> > > Hi, pulsar community:
> > >
> > > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > > partitioned topic or multi-topics`
> > >
> > > discuss thread:
> > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > >
> > > PIP: https://github.com/apache/pulsar/issues/18182
> > >
> > > Thanks,
> > > bo
> > >
>
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by Haiting Jiang <ji...@gmail.com>.
+1
Haiting
On Thu, Oct 27, 2022 at 5:55 PM Yubiao Feng
<yu...@streamnative.io.invalid> wrote:
>
> +1
>
> On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
>
> > Hi, pulsar community:
> >
> > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > partitioned topic or multi-topics`
> >
> > discuss thread:
> > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> >
> > PIP: https://github.com/apache/pulsar/issues/18182
> >
> > Thanks,
> > bo
> >
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by Yubiao Feng <yu...@streamnative.io.INVALID>.
+1
On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
> Hi, pulsar community:
>
> I start a PIP about `User-friendly acknowledgeCumulative API on a
> partitioned topic or multi-topics`
>
> discuss thread:
> https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
>
> PIP: https://github.com/apache/pulsar/issues/18182
>
> Thanks,
> bo
>
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by Yunze Xu <yz...@streamnative.io.INVALID>.
+1
Thanks,
Yunze
On Wed, Oct 26, 2022 at 12:06 PM Qiang Huang <qi...@gmail.com> wrote:
>
> +1
>
> Enrico Olivelli <eo...@gmail.com> 于2022年10月25日周二 22:32写道：
>
> > +1
> >
> > awesome work
> >
> > Enrico
> >
> > Il giorno mar 25 ott 2022 alle ore 16:28 PengHui Li
> > <pe...@apache.org> ha scritto:
> > >
> > > +1
> > >
> > > Penghui
> > >
> > > On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
> > >
> > > > Hi, pulsar community:
> > > >
> > > > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > > > partitioned topic or multi-topics`
> > > >
> > > > discuss thread:
> > > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > >
> > > > PIP: https://github.com/apache/pulsar/issues/18182
> > > >
> > > > Thanks,
> > > > bo
> > > >
> >
>
>
> --
> BR,
> Qiang Huang
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by Qiang Huang <qi...@gmail.com>.
+1
Enrico Olivelli <eo...@gmail.com> 于2022年10月25日周二 22:32写道：
> +1
>
> awesome work
>
> Enrico
>
> Il giorno mar 25 ott 2022 alle ore 16:28 PengHui Li
> <pe...@apache.org> ha scritto:
> >
> > +1
> >
> > Penghui
> >
> > On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
> >
> > > Hi, pulsar community:
> > >
> > > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > > partitioned topic or multi-topics`
> > >
> > > discuss thread:
> > > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > >
> > > PIP: https://github.com/apache/pulsar/issues/18182
> > >
> > > Thanks,
> > > bo
> > >
>
--
BR,
Qiang Huang
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by Enrico Olivelli <eo...@gmail.com>.
+1
awesome work
Enrico
Il giorno mar 25 ott 2022 alle ore 16:28 PengHui Li
<pe...@apache.org> ha scritto:
>
> +1
>
> Penghui
>
> On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
>
> > Hi, pulsar community:
> >
> > I start a PIP about `User-friendly acknowledgeCumulative API on a
> > partitioned topic or multi-topics`
> >
> > discuss thread:
> > https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> >
> > PIP: https://github.com/apache/pulsar/issues/18182
> >
> > Thanks,
> > bo
> >
Re: [DISCUSS] PIP-218: Consumer batchReceive() single partition every receive
Posted by PengHui Li <pe...@apache.org>.
+1
Penghui
On Tue, Oct 25, 2022 at 12:25 PM 丛搏 <bo...@apache.org> wrote:
> Hi, pulsar community:
>
> I start a PIP about `User-friendly acknowledgeCumulative API on a
> partitioned topic or multi-topics`
>
> discuss thread:
> https://lists.apache.org/thread/30rwksz4gmvgspkgcfsk708sgb1n7vbo
> https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
>
> PIP: https://github.com/apache/pulsar/issues/18182
>
> Thanks,
> bo
>