You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Haiting Jiang <ji...@apache.org> on 2021/12/20 07:37:00 UTC
[DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
Hi Pulsar Community,
I discovered a bug that chunk messages producing fails if topic level maxMessageSize is set [1]. The root cause of this issue is because chunk message is using broker level maxMessageSize as chunk size. And topic level maxMessageSize is always <= broker level maxMessageSize. So once it is set, the on-going chunk message producing fails.
## Proposed changes
I would like to fix this by just skipping topic level maxMessageSize check in org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic level maxMessageSize is introduced in [2], for the purpose of "easier to plan resource quotas for client allocation". And IMO this change will not bring further complex into this.
## Alternative
Add a client side topic level maxMessageSize and keep it synced with broker.
Required changes:
- [client] Add a new field org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this client side topic level maxMessageSize.
- [PulsarApi.proto] Add a new field maxMessageSize in the CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
- [PulsarApi.proto] Add a new Command like CommandUpdateClientPolicy{producerId, maxMessageSize} to update ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
Further more, some other data consistency issues need be handled very carefully when maxMessageSize is updated.
This alternative is complex but can also solve other topic level maxMessageSize issue [3] when batching is enabled (non-batching case is solved with PR [4]).
Any suggestions or other use cases of topic level maxMessageSize will be appreciated.
Thanks,
Haiting Jiang
[1] https://github.com/apache/pulsar/issues/13360
[2] https://github.com/apache/pulsar/pull/8732
[3] https://github.com/apache/pulsar/issues/12958
[4] https://github.com/apache/pulsar/pull/13147
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
Posted by Enrico Olivelli <eo...@gmail.com>.
It looks loke you already prepared a awesome PIP!
Please copy your proposal on a PIP.
I am +1
Enrico
Il Mar 28 Dic 2021, 04:27 Zike Yang <zk...@streamnative.io.invalid> ha
scritto:
> + 1 for skipping the topic level message size check for the chunk message
> in the broker.
>
> On Tue, Dec 28, 2021 at 9:14 AM guo jiwei <te...@apache.org> wrote:
>
> > +1
> >
> >
> > Regards
> > Jiwei Guo (Tboy)
> >
> > On Tue, Dec 28, 2021 at 5:44 AM 陳智弘 <th...@gmail.com> wrote:
> > >
> > > +1
> > >
> > > Skip the topic level setting is better
> > >
> > > Hang Chen <ch...@apache.org> 於 2021年12月27日 週一 22:09 寫道：
> > >
> > > > +1
> > > >
> > > > We'd better skip the topic level maxMessageSize check for chunk
> > messages.
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > PengHui Li <pe...@apache.org> 于2021年12月27日周一 22:07写道：
> > > > >
> > > > > +1,
> > > > >
> > > > > We can only skip the topic level messages size check for the chunk
> > > > message.
> > > > >
> > > > > Regards,
> > > > > Penghui
> > > > >
> > > > > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <
> > jianghaiting@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Pulsar Community,
> > > > > >
> > > > > > I discovered a bug that chunk messages producing fails if topic
> > level
> > > > > > maxMessageSize is set [1]. The root cause of this issue is
> because
> > > > chunk
> > > > > > message is using broker level maxMessageSize as chunk size. And
> > topic
> > > > level
> > > > > > maxMessageSize is always <= broker level maxMessageSize. So once
> > it is
> > > > set,
> > > > > > the on-going chunk message producing fails.
> > > > > >
> > > > > > ## Proposed changes
> > > > > > I would like to fix this by just skipping topic level
> > maxMessageSize
> > > > check
> > > > > > in
> > > > > >
> > > >
> >
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > > > > level maxMessageSize is introduced in [2], for the purpose of
> > "easier
> > > > to
> > > > > > plan resource quotas for client allocation". And IMO this change
> > will
> > > > not
> > > > > > bring further complex into this.
> > > > > >
> > > > > > ## Alternative
> > > > > > Add a client side topic level maxMessageSize and keep it synced
> > with
> > > > > > broker.
> > > > > >
> > > > > > Required changes:
> > > > > > - [client] Add a new field
> > > > > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to
> store
> > this
> > > > > > client side topic level maxMessageSize.
> > > > > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > > > > CommandProducerSuccess for the initial value of
> > > > ProducerBase#maxMessageSize
> > > > > > - [PulsarApi.proto] Add a new Command like
> > > > > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > > > > ProducerBase#maxMessageSize when topic level maxMessageSize is
> > updated.
> > > > > > Further more, some other data consistency issues need be handled
> > very
> > > > > > carefully when maxMessageSize is updated.
> > > > > > This alternative is complex but can also solve other topic level
> > > > > > maxMessageSize issue [3] when batching is enabled (non-batching
> > case is
> > > > > > solved with PR [4]).
> > > > > >
> > > > > > Any suggestions or other use cases of topic level maxMessageSize
> > will
> > > > be
> > > > > > appreciated.
> > > > > >
> > > > > > Thanks,
> > > > > > Haiting Jiang
> > > > > >
> > > > > > [1] https://github.com/apache/pulsar/issues/13360
> > > > > > [2] https://github.com/apache/pulsar/pull/8732
> > > > > > [3] https://github.com/apache/pulsar/issues/12958
> > > > > > [4] https://github.com/apache/pulsar/pull/13147
> > > > > >
> > > >
> >
>
>
> --
> Zike Yang
>
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
Posted by Zike Yang <zk...@streamnative.io.INVALID>.
+ 1 for skipping the topic level message size check for the chunk message
in the broker.
On Tue, Dec 28, 2021 at 9:14 AM guo jiwei <te...@apache.org> wrote:
> +1
>
>
> Regards
> Jiwei Guo (Tboy)
>
> On Tue, Dec 28, 2021 at 5:44 AM 陳智弘 <th...@gmail.com> wrote:
> >
> > +1
> >
> > Skip the topic level setting is better
> >
> > Hang Chen <ch...@apache.org> 於 2021年12月27日 週一 22:09 寫道：
> >
> > > +1
> > >
> > > We'd better skip the topic level maxMessageSize check for chunk
> messages.
> > >
> > > Best,
> > > Hang
> > >
> > > PengHui Li <pe...@apache.org> 于2021年12月27日周一 22:07写道：
> > > >
> > > > +1,
> > > >
> > > > We can only skip the topic level messages size check for the chunk
> > > message.
> > > >
> > > > Regards,
> > > > Penghui
> > > >
> > > > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <
> jianghaiting@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Pulsar Community,
> > > > >
> > > > > I discovered a bug that chunk messages producing fails if topic
> level
> > > > > maxMessageSize is set [1]. The root cause of this issue is because
> > > chunk
> > > > > message is using broker level maxMessageSize as chunk size. And
> topic
> > > level
> > > > > maxMessageSize is always <= broker level maxMessageSize. So once
> it is
> > > set,
> > > > > the on-going chunk message producing fails.
> > > > >
> > > > > ## Proposed changes
> > > > > I would like to fix this by just skipping topic level
> maxMessageSize
> > > check
> > > > > in
> > > > >
> > >
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > > > level maxMessageSize is introduced in [2], for the purpose of
> "easier
> > > to
> > > > > plan resource quotas for client allocation". And IMO this change
> will
> > > not
> > > > > bring further complex into this.
> > > > >
> > > > > ## Alternative
> > > > > Add a client side topic level maxMessageSize and keep it synced
> with
> > > > > broker.
> > > > >
> > > > > Required changes:
> > > > > - [client] Add a new field
> > > > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store
> this
> > > > > client side topic level maxMessageSize.
> > > > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > > > CommandProducerSuccess for the initial value of
> > > ProducerBase#maxMessageSize
> > > > > - [PulsarApi.proto] Add a new Command like
> > > > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > > > ProducerBase#maxMessageSize when topic level maxMessageSize is
> updated.
> > > > > Further more, some other data consistency issues need be handled
> very
> > > > > carefully when maxMessageSize is updated.
> > > > > This alternative is complex but can also solve other topic level
> > > > > maxMessageSize issue [3] when batching is enabled (non-batching
> case is
> > > > > solved with PR [4]).
> > > > >
> > > > > Any suggestions or other use cases of topic level maxMessageSize
> will
> > > be
> > > > > appreciated.
> > > > >
> > > > > Thanks,
> > > > > Haiting Jiang
> > > > >
> > > > > [1] https://github.com/apache/pulsar/issues/13360
> > > > > [2] https://github.com/apache/pulsar/pull/8732
> > > > > [3] https://github.com/apache/pulsar/issues/12958
> > > > > [4] https://github.com/apache/pulsar/pull/13147
> > > > >
> > >
>
--
Zike Yang
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
Posted by guo jiwei <te...@apache.org>.
+1
Regards
Jiwei Guo (Tboy)
On Tue, Dec 28, 2021 at 5:44 AM 陳智弘 <th...@gmail.com> wrote:
>
> +1
>
> Skip the topic level setting is better
>
> Hang Chen <ch...@apache.org> 於 2021年12月27日 週一 22:09 寫道：
>
> > +1
> >
> > We'd better skip the topic level maxMessageSize check for chunk messages.
> >
> > Best,
> > Hang
> >
> > PengHui Li <pe...@apache.org> 于2021年12月27日周一 22:07写道：
> > >
> > > +1,
> > >
> > > We can only skip the topic level messages size check for the chunk
> > message.
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <ji...@apache.org>
> > > wrote:
> > >
> > > > Hi Pulsar Community,
> > > >
> > > > I discovered a bug that chunk messages producing fails if topic level
> > > > maxMessageSize is set [1]. The root cause of this issue is because
> > chunk
> > > > message is using broker level maxMessageSize as chunk size. And topic
> > level
> > > > maxMessageSize is always <= broker level maxMessageSize. So once it is
> > set,
> > > > the on-going chunk message producing fails.
> > > >
> > > > ## Proposed changes
> > > > I would like to fix this by just skipping topic level maxMessageSize
> > check
> > > > in
> > > >
> > org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > > level maxMessageSize is introduced in [2], for the purpose of "easier
> > to
> > > > plan resource quotas for client allocation". And IMO this change will
> > not
> > > > bring further complex into this.
> > > >
> > > > ## Alternative
> > > > Add a client side topic level maxMessageSize and keep it synced with
> > > > broker.
> > > >
> > > > Required changes:
> > > > - [client] Add a new field
> > > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> > > > client side topic level maxMessageSize.
> > > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > > CommandProducerSuccess for the initial value of
> > ProducerBase#maxMessageSize
> > > > - [PulsarApi.proto] Add a new Command like
> > > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > > ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > > > Further more, some other data consistency issues need be handled very
> > > > carefully when maxMessageSize is updated.
> > > > This alternative is complex but can also solve other topic level
> > > > maxMessageSize issue [3] when batching is enabled (non-batching case is
> > > > solved with PR [4]).
> > > >
> > > > Any suggestions or other use cases of topic level maxMessageSize will
> > be
> > > > appreciated.
> > > >
> > > > Thanks,
> > > > Haiting Jiang
> > > >
> > > > [1] https://github.com/apache/pulsar/issues/13360
> > > > [2] https://github.com/apache/pulsar/pull/8732
> > > > [3] https://github.com/apache/pulsar/issues/12958
> > > > [4] https://github.com/apache/pulsar/pull/13147
> > > >
> >
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
Posted by 陳智弘 <th...@gmail.com>.
+1
Skip the topic level setting is better
Hang Chen <ch...@apache.org> 於 2021年12月27日 週一 22:09 寫道：
> +1
>
> We'd better skip the topic level maxMessageSize check for chunk messages.
>
> Best,
> Hang
>
> PengHui Li <pe...@apache.org> 于2021年12月27日周一 22:07写道：
> >
> > +1,
> >
> > We can only skip the topic level messages size check for the chunk
> message.
> >
> > Regards,
> > Penghui
> >
> > On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <ji...@apache.org>
> > wrote:
> >
> > > Hi Pulsar Community,
> > >
> > > I discovered a bug that chunk messages producing fails if topic level
> > > maxMessageSize is set [1]. The root cause of this issue is because
> chunk
> > > message is using broker level maxMessageSize as chunk size. And topic
> level
> > > maxMessageSize is always <= broker level maxMessageSize. So once it is
> set,
> > > the on-going chunk message producing fails.
> > >
> > > ## Proposed changes
> > > I would like to fix this by just skipping topic level maxMessageSize
> check
> > > in
> > >
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > > level maxMessageSize is introduced in [2], for the purpose of "easier
> to
> > > plan resource quotas for client allocation". And IMO this change will
> not
> > > bring further complex into this.
> > >
> > > ## Alternative
> > > Add a client side topic level maxMessageSize and keep it synced with
> > > broker.
> > >
> > > Required changes:
> > > - [client] Add a new field
> > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> > > client side topic level maxMessageSize.
> > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > CommandProducerSuccess for the initial value of
> ProducerBase#maxMessageSize
> > > - [PulsarApi.proto] Add a new Command like
> > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > > Further more, some other data consistency issues need be handled very
> > > carefully when maxMessageSize is updated.
> > > This alternative is complex but can also solve other topic level
> > > maxMessageSize issue [3] when batching is enabled (non-batching case is
> > > solved with PR [4]).
> > >
> > > Any suggestions or other use cases of topic level maxMessageSize will
> be
> > > appreciated.
> > >
> > > Thanks,
> > > Haiting Jiang
> > >
> > > [1] https://github.com/apache/pulsar/issues/13360
> > > [2] https://github.com/apache/pulsar/pull/8732
> > > [3] https://github.com/apache/pulsar/issues/12958
> > > [4] https://github.com/apache/pulsar/pull/13147
> > >
>
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
Posted by Hang Chen <ch...@apache.org>.
+1
We'd better skip the topic level maxMessageSize check for chunk messages.
Best,
Hang
PengHui Li <pe...@apache.org> 于2021年12月27日周一 22:07写道：
>
> +1,
>
> We can only skip the topic level messages size check for the chunk message.
>
> Regards,
> Penghui
>
> On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <ji...@apache.org>
> wrote:
>
> > Hi Pulsar Community,
> >
> > I discovered a bug that chunk messages producing fails if topic level
> > maxMessageSize is set [1]. The root cause of this issue is because chunk
> > message is using broker level maxMessageSize as chunk size. And topic level
> > maxMessageSize is always <= broker level maxMessageSize. So once it is set,
> > the on-going chunk message producing fails.
> >
> > ## Proposed changes
> > I would like to fix this by just skipping topic level maxMessageSize check
> > in
> > org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> > level maxMessageSize is introduced in [2], for the purpose of "easier to
> > plan resource quotas for client allocation". And IMO this change will not
> > bring further complex into this.
> >
> > ## Alternative
> > Add a client side topic level maxMessageSize and keep it synced with
> > broker.
> >
> > Required changes:
> > - [client] Add a new field
> > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> > client side topic level maxMessageSize.
> > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> > - [PulsarApi.proto] Add a new Command like
> > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > Further more, some other data consistency issues need be handled very
> > carefully when maxMessageSize is updated.
> > This alternative is complex but can also solve other topic level
> > maxMessageSize issue [3] when batching is enabled (non-batching case is
> > solved with PR [4]).
> >
> > Any suggestions or other use cases of topic level maxMessageSize will be
> > appreciated.
> >
> > Thanks,
> > Haiting Jiang
> >
> > [1] https://github.com/apache/pulsar/issues/13360
> > [2] https://github.com/apache/pulsar/pull/8732
> > [3] https://github.com/apache/pulsar/issues/12958
> > [4] https://github.com/apache/pulsar/pull/13147
> >
Re: [DISCUSSION] Produce chunk messages failed when topic level maxMessageSize is set
Posted by PengHui Li <pe...@apache.org>.
+1,
We can only skip the topic level messages size check for the chunk message.
Regards,
Penghui
On Mon, Dec 20, 2021 at 3:37 PM Haiting Jiang <ji...@apache.org>
wrote:
> Hi Pulsar Community,
>
> I discovered a bug that chunk messages producing fails if topic level
> maxMessageSize is set [1]. The root cause of this issue is because chunk
> message is using broker level maxMessageSize as chunk size. And topic level
> maxMessageSize is always <= broker level maxMessageSize. So once it is set,
> the on-going chunk message producing fails.
>
> ## Proposed changes
> I would like to fix this by just skipping topic level maxMessageSize check
> in
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.Topic
> level maxMessageSize is introduced in [2], for the purpose of "easier to
> plan resource quotas for client allocation". And IMO this change will not
> bring further complex into this.
>
> ## Alternative
> Add a client side topic level maxMessageSize and keep it synced with
> broker.
>
> Required changes:
> - [client] Add a new field
> org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> client side topic level maxMessageSize.
> - [PulsarApi.proto] Add a new field maxMessageSize in the
> CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> - [PulsarApi.proto] Add a new Command like
> CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> Further more, some other data consistency issues need be handled very
> carefully when maxMessageSize is updated.
> This alternative is complex but can also solve other topic level
> maxMessageSize issue [3] when batching is enabled (non-batching case is
> solved with PR [4]).
>
> Any suggestions or other use cases of topic level maxMessageSize will be
> appreciated.
>
> Thanks,
> Haiting Jiang
>
> [1] https://github.com/apache/pulsar/issues/13360
> [2] https://github.com/apache/pulsar/pull/8732
> [3] https://github.com/apache/pulsar/issues/12958
> [4] https://github.com/apache/pulsar/pull/13147
>