Re: [DISCUSS] Why not split `memoryLimit` into `consumerMemoryLimit ` and `producerMemoryLimit `

[Zike Yang <zi...@apache.org>]

Hi, JIaqi

Thanks for initiating this discussion.

Do you have any cases that need separate limit control for producers/consumers?

We have max pending queue size and max message size for the producer,
and receiver queue size for the consumer. Although they hardly control
the limit as the byte-granular, they seem to be sufficient for the
current needs.

If there is a strong case, I think we can consider this feature.

Thanks,
Zike Yang


Zike Yang

On Tue, Feb 28, 2023 at 10:25 AM Jiaqi Shen <gl...@gmail.com> wrote:
>
> Context.
> - PIP-74
> https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits
> - PR-8965 https://github.com/apache/pulsar/pull/8965
> - PR-15216 https://github.com/apache/pulsar/pull/15216
>
> Hello, community:
>
> There are some questions about PIP-74 I want to figure out.
>
> PIP-74 and its implementation specifie how to limit *client* memory. But in
> our scenario, the *client* is usually reused. It is more intuitive to limit
> a single *producer* or *consumer*. So why not let the producer/consumer
> have their own MemoryLimitController? And shoud we split the
> memoryLimit setting
> into consumerMemoryLimit and producerMemoryLimit?  Are there any other
> considerations to limit *client* memory?
>
> If you know why it needs to be designed like this, please leave your
> comment. Thanks!
>
> Thanks,
> Jiaqi Shen

Re: [DISCUSS] Why not split `memoryLimit` into `consumerMemoryLimit ` and `producerMemoryLimit `

[ZhangJian He <sh...@gmail.com>]

Producers and Consumers share many things including memory limit. In some
strong cases, maybe use two pulsar clients, one for producer and other for
consumer, might be a good choice.

Thanks
ZhangJian He


On Wed, 1 Mar 2023 at 17:15, Zike Yang <zi...@apache.org> wrote:

> Hi, JIaqi
>
> Thanks for initiating this discussion.
>
> Do you have any cases that need separate limit control for
> producers/consumers?
>
> We have max pending queue size and max message size for the producer,
> and receiver queue size for the consumer. Although they hardly control
> the limit as the byte-granular, they seem to be sufficient for the
> current needs.
>
> If there is a strong case, I think we can consider this feature.
>
> Thanks,
> Zike Yang
>
>
> Zike Yang
>
> On Tue, Feb 28, 2023 at 10:25 AM Jiaqi Shen <gl...@gmail.com>
> wrote:
> >
> > Context.
> > - PIP-74
> >
> https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits
> > - PR-8965 https://github.com/apache/pulsar/pull/8965
> > - PR-15216 https://github.com/apache/pulsar/pull/15216
> >
> > Hello, community:
> >
> > There are some questions about PIP-74 I want to figure out.
> >
> > PIP-74 and its implementation specifie how to limit *client* memory. But
> in
> > our scenario, the *client* is usually reused. It is more intuitive to
> limit
> > a single *producer* or *consumer*. So why not let the producer/consumer
> > have their own MemoryLimitController? And shoud we split the
> > memoryLimit setting
> > into consumerMemoryLimit and producerMemoryLimit?  Are there any other
> > considerations to limit *client* memory?
> >
> > If you know why it needs to be designed like this, please leave your
> > comment. Thanks!
> >
> > Thanks,
> > Jiaqi Shen
>