You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pulsar.apache.org by Haiting Jiang <ji...@apache.org> on 2021/12/29 02:29:21 UTC

[VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set

This is the voting thread for PIP-131. It will stay open for at least 48h.

https://github.com/apache/pulsar/issues/13544

The discussion thread is https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf

## Motivation

Currently, chunk messages producing fails if topic level maxMessageSize is set [1]. The root cause of this issue is because chunk message is using broker level maxMessageSize as chunk size. And topic level maxMessageSize is always <= broker level maxMessageSize. So once it is set, the on-going chunk message producing fails.

## Goal

Resolve topic level maxMessageSize compatibility issue with chunking messages.

## Implementation

Current best solution would be just skipping topic level maxMessageSize check in org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.
Topic level maxMessageSize is introduced in [2], for the purpose of "easier to plan resource quotas for client allocation". And IMO this change will not bring further complex into this.

## Reject Alternatives

Add a client side topic level maxMessageSize and keep it synced with broker.

Required changes:
- [client] Add a new field org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this client side topic level maxMessageSize.
- [PulsarApi.proto] Add a new field maxMessageSize in the CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
- [PulsarApi.proto] Add a new Command like CommandUpdateClientPolicy{producerId, maxMessageSize} to update ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
Further more, some other data consistency issues need be handled very carefully when maxMessageSize is updated.
This alternative is complex but can also solve other topic level maxMessageSize issue [3] when batching is enabled (non-batching case is solved with PR [4]).

[1] https://github.com/apache/pulsar/issues/13360
[2] https://github.com/apache/pulsar/pull/8732
[3] https://github.com/apache/pulsar/issues/12958
[4] https://github.com/apache/pulsar/pull/13147

Thanks,
Haiting Jiang

Re: [VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set

Posted by Lan Liang <li...@163.com>.

Best Regards,
Lan Liang
On 12/29/2021 10:29，Haiting Jiang<ji...@apache.org> wrote：
This is the voting thread for PIP-131. It will stay open for at least 48h.

https://github.com/apache/pulsar/issues/13544

The discussion thread is https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf

## Motivation

## Goal

Resolve topic level maxMessageSize compatibility issue with chunking messages.

## Implementation

## Reject Alternatives

Add a client side topic level maxMessageSize and keep it synced with broker.

[1] https://github.com/apache/pulsar/issues/13360
[2] https://github.com/apache/pulsar/pull/8732
[3] https://github.com/apache/pulsar/issues/12958
[4] https://github.com/apache/pulsar/pull/13147

Thanks,
Haiting Jiang

Re: [VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set

Posted by Hang Chen <ch...@apache.org>.

+1

Best,
Hang

Zike Yang <zk...@streamnative.io.invalid> 于2021年12月29日周三 15:04写道：
>
> +1
>
> Thanks,
> Zike
>
> On Wed, Dec 29, 2021 at 11:26 AM PengHui Li <pe...@apache.org> wrote:
> >
> > +1
> >
> > Thanks,
> > Penghui
> >
> > On Wed, Dec 29, 2021 at 10:29 AM Haiting Jiang <ji...@apache.org>
> > wrote:
> >
> > > This is the voting thread for PIP-131. It will stay open for at least 48h.
> > >
> > > https://github.com/apache/pulsar/issues/13544
> > >
> > > The discussion thread is
> > > https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf
> > >
> > > ## Motivation
> > >
> > > Currently, chunk messages producing fails if topic level maxMessageSize is
> > > set [1]. The root cause of this issue is because chunk message is using
> > > broker level maxMessageSize as chunk size. And topic level maxMessageSize
> > > is always <= broker level maxMessageSize. So once it is set, the on-going
> > > chunk message producing fails.
> > >
> > > ## Goal
> > >
> > > Resolve topic level maxMessageSize compatibility issue with chunking
> > > messages.
> > >
> > > ## Implementation
> > >
> > > Current best solution would be just skipping topic level maxMessageSize
> > > check in
> > > org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.
> > > Topic level maxMessageSize is introduced in [2], for the purpose of
> > > "easier to plan resource quotas for client allocation". And IMO this change
> > > will not bring further complex into this.
> > >
> > > ## Reject Alternatives
> > >
> > > Add a client side topic level maxMessageSize and keep it synced with
> > > broker.
> > >
> > > Required changes:
> > > - [client] Add a new field
> > > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> > > client side topic level maxMessageSize.
> > > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > > CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> > > - [PulsarApi.proto] Add a new Command like
> > > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > > ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > > Further more, some other data consistency issues need be handled very
> > > carefully when maxMessageSize is updated.
> > > This alternative is complex but can also solve other topic level
> > > maxMessageSize issue [3] when batching is enabled (non-batching case is
> > > solved with PR [4]).
> > >
> > > [1] https://github.com/apache/pulsar/issues/13360
> > > [2] https://github.com/apache/pulsar/pull/8732
> > > [3] https://github.com/apache/pulsar/issues/12958
> > > [4] https://github.com/apache/pulsar/pull/13147
> > >
> > > Thanks,
> > > Haiting Jiang
> > >
>
>
>
> --
> Zike Yang

Re: [VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set

Posted by Zike Yang <zk...@streamnative.io.INVALID>.

+1

Thanks,
Zike

On Wed, Dec 29, 2021 at 11:26 AM PengHui Li <pe...@apache.org> wrote:
>
> +1
>
> Thanks,
> Penghui
>
> On Wed, Dec 29, 2021 at 10:29 AM Haiting Jiang <ji...@apache.org>
> wrote:
>
> > This is the voting thread for PIP-131. It will stay open for at least 48h.
> >
> > https://github.com/apache/pulsar/issues/13544
> >
> > The discussion thread is
> > https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf
> >
> > ## Motivation
> >
> > Currently, chunk messages producing fails if topic level maxMessageSize is
> > set [1]. The root cause of this issue is because chunk message is using
> > broker level maxMessageSize as chunk size. And topic level maxMessageSize
> > is always <= broker level maxMessageSize. So once it is set, the on-going
> > chunk message producing fails.
> >
> > ## Goal
> >
> > Resolve topic level maxMessageSize compatibility issue with chunking
> > messages.
> >
> > ## Implementation
> >
> > Current best solution would be just skipping topic level maxMessageSize
> > check in
> > org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.
> > Topic level maxMessageSize is introduced in [2], for the purpose of
> > "easier to plan resource quotas for client allocation". And IMO this change
> > will not bring further complex into this.
> >
> > ## Reject Alternatives
> >
> > Add a client side topic level maxMessageSize and keep it synced with
> > broker.
> >
> > Required changes:
> > - [client] Add a new field
> > org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> > client side topic level maxMessageSize.
> > - [PulsarApi.proto] Add a new field maxMessageSize in the
> > CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> > - [PulsarApi.proto] Add a new Command like
> > CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> > ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > Further more, some other data consistency issues need be handled very
> > carefully when maxMessageSize is updated.
> > This alternative is complex but can also solve other topic level
> > maxMessageSize issue [3] when batching is enabled (non-batching case is
> > solved with PR [4]).
> >
> > [1] https://github.com/apache/pulsar/issues/13360
> > [2] https://github.com/apache/pulsar/pull/8732
> > [3] https://github.com/apache/pulsar/issues/12958
> > [4] https://github.com/apache/pulsar/pull/13147
> >
> > Thanks,
> > Haiting Jiang
> >



-- 
Zike Yang

Re: [VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set

Posted by PengHui Li <pe...@apache.org>.

+1

Thanks,
Penghui

On Wed, Dec 29, 2021 at 10:29 AM Haiting Jiang <ji...@apache.org>
wrote:

> This is the voting thread for PIP-131. It will stay open for at least 48h.
>
> https://github.com/apache/pulsar/issues/13544
>
> The discussion thread is
> https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf
>
> ## Motivation
>
> Currently, chunk messages producing fails if topic level maxMessageSize is
> set [1]. The root cause of this issue is because chunk message is using
> broker level maxMessageSize as chunk size. And topic level maxMessageSize
> is always <= broker level maxMessageSize. So once it is set, the on-going
> chunk message producing fails.
>
> ## Goal
>
> Resolve topic level maxMessageSize compatibility issue with chunking
> messages.
>
> ## Implementation
>
> Current best solution would be just skipping topic level maxMessageSize
> check in
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.
> Topic level maxMessageSize is introduced in [2], for the purpose of
> "easier to plan resource quotas for client allocation". And IMO this change
> will not bring further complex into this.
>
> ## Reject Alternatives
>
> Add a client side topic level maxMessageSize and keep it synced with
> broker.
>
> Required changes:
> - [client] Add a new field
> org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> client side topic level maxMessageSize.
> - [PulsarApi.proto] Add a new field maxMessageSize in the
> CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> - [PulsarApi.proto] Add a new Command like
> CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> Further more, some other data consistency issues need be handled very
> carefully when maxMessageSize is updated.
> This alternative is complex but can also solve other topic level
> maxMessageSize issue [3] when batching is enabled (non-batching case is
> solved with PR [4]).
>
> [1] https://github.com/apache/pulsar/issues/13360
> [2] https://github.com/apache/pulsar/pull/8732
> [3] https://github.com/apache/pulsar/issues/12958
> [4] https://github.com/apache/pulsar/pull/13147
>
> Thanks,
> Haiting Jiang
>

Re: [VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set

Posted by PengHui Li <pe...@apache.org>.

Close the VOTE with 3 (+1) + 2 bindings and 0 (-1) bindings

Thanks,
Penghui

On Tue, Jan 18, 2022 at 10:30 AM Lin Lin <li...@apache.org> wrote:

> +1
>
> On 2021/12/29 02:29:21 Haiting Jiang wrote:
> > This is the voting thread for PIP-131. It will stay open for at least
> 48h.
> >
> > https://github.com/apache/pulsar/issues/13544
> >
> > The discussion thread is
> https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf
> >
> > ## Motivation
> >
> > Currently, chunk messages producing fails if topic level maxMessageSize
> is set [1]. The root cause of this issue is because chunk message is using
> broker level maxMessageSize as chunk size. And topic level maxMessageSize
> is always <= broker level maxMessageSize. So once it is set, the on-going
> chunk message producing fails.
> >
> > ## Goal
> >
> > Resolve topic level maxMessageSize compatibility issue with chunking
> messages.
> >
> > ## Implementation
> >
> > Current best solution would be just skipping topic level maxMessageSize
> check in
> org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize.
> > Topic level maxMessageSize is introduced in [2], for the purpose of
> "easier to plan resource quotas for client allocation". And IMO this change
> will not bring further complex into this.
> >
> > ## Reject Alternatives
> >
> > Add a client side topic level maxMessageSize and keep it synced with
> broker.
> >
> > Required changes:
> > - [client] Add a new field
> org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this
> client side topic level maxMessageSize.
> > - [PulsarApi.proto] Add a new field maxMessageSize in the
> CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> > - [PulsarApi.proto] Add a new Command like
> CommandUpdateClientPolicy{producerId, maxMessageSize} to update
> ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> > Further more, some other data consistency issues need be handled very
> carefully when maxMessageSize is updated.
> > This alternative is complex but can also solve other topic level
> maxMessageSize issue [3] when batching is enabled (non-batching case is
> solved with PR [4]).
> >
> > [1] https://github.com/apache/pulsar/issues/13360
> > [2] https://github.com/apache/pulsar/pull/8732
> > [3] https://github.com/apache/pulsar/issues/12958
> > [4] https://github.com/apache/pulsar/pull/13147
> >
> > Thanks,
> > Haiting Jiang
> >
>

Re: [VOTE] PIP 131: Resolve produce chunk messages failed when topic level maxMessageSize is set

Posted by Lin Lin <li...@apache.org>.

+1

On 2021/12/29 02:29:21 Haiting Jiang wrote:
> This is the voting thread for PIP-131. It will stay open for at least 48h.
> 
> https://github.com/apache/pulsar/issues/13544
> 
> The discussion thread is https://lists.apache.org/thread/c63d9s73j9x1m3dkqr3r38gyp8s7cwzf
> 
> ## Motivation
> 
> Currently, chunk messages producing fails if topic level maxMessageSize is set [1]. The root cause of this issue is because chunk message is using broker level maxMessageSize as chunk size. And topic level maxMessageSize is always <= broker level maxMessageSize. So once it is set, the on-going chunk message producing fails.
> 
> ## Goal
> 
> Resolve topic level maxMessageSize compatibility issue with chunking messages.
> 
> ## Implementation
> 
> Current best solution would be just skipping topic level maxMessageSize check in org.apache.pulsar.broker.service.AbstractTopic#isExceedMaximumMessageSize. 
> Topic level maxMessageSize is introduced in [2], for the purpose of "easier to plan resource quotas for client allocation". And IMO this change will not bring further complex into this.
> 
> ## Reject Alternatives
> 
> Add a client side topic level maxMessageSize and keep it synced with broker.
> 
> Required changes:
> - [client] Add a new field org.apache.pulsar.client.impl.ProducerBase#maxMessageSize to store this client side topic level maxMessageSize.
> - [PulsarApi.proto] Add a new field maxMessageSize in the CommandProducerSuccess for the initial value of ProducerBase#maxMessageSize
> - [PulsarApi.proto] Add a new Command like CommandUpdateClientPolicy{producerId, maxMessageSize} to update ProducerBase#maxMessageSize when topic level maxMessageSize is updated.
> Further more, some other data consistency issues need be handled very carefully when maxMessageSize is updated.
> This alternative is complex but can also solve other topic level maxMessageSize issue [3] when batching is enabled (non-batching case is solved with PR [4]).
> 
> [1] https://github.com/apache/pulsar/issues/13360
> [2] https://github.com/apache/pulsar/pull/8732
> [3] https://github.com/apache/pulsar/issues/12958
> [4] https://github.com/apache/pulsar/pull/13147
> 
> Thanks,
> Haiting Jiang
>