[DISCUSS] Why not split `memoryLimit` into `consumerMemoryLimit ` and `producerMemoryLimit `

[Jiaqi Shen <gl...@gmail.com>]

Context.
- PIP-74
https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits
- PR-8965 https://github.com/apache/pulsar/pull/8965
- PR-15216 https://github.com/apache/pulsar/pull/15216

Hello, community:

There are some questions about PIP-74 I want to figure out.

PIP-74 and its implementation specifie how to limit *client* memory. But in
our scenario, the *client* is usually reused. It is more intuitive to limit
a single *producer* or *consumer*. So why not let the producer/consumer
have their own MemoryLimitController? And shoud we split the
memoryLimit setting
into consumerMemoryLimit and producerMemoryLimit?  Are there any other
considerations to limit *client* memory?

If you know why it needs to be designed like this, please leave your
comment. Thanks!

Thanks,
Jiaqi Shen

Re: [DISCUSS] Why not split `memoryLimit` into `consumerMemoryLimit ` and `producerMemoryLimit `

[ZhangJian He <sh...@gmail.com>]

Producers and Consumers share many things including memory limit. In some
strong cases, maybe use two pulsar clients, one for producer and other for
consumer, might be a good choice.

Thanks
ZhangJian He


On Wed, 1 Mar 2023 at 17:15, Zike Yang <zi...@apache.org> wrote:

> Hi, JIaqi
>
> Thanks for initiating this discussion.
>
> Do you have any cases that need separate limit control for
> producers/consumers?
>
> We have max pending queue size and max message size for the producer,
> and receiver queue size for the consumer. Although they hardly control
> the limit as the byte-granular, they seem to be sufficient for the
> current needs.
>
> If there is a strong case, I think we can consider this feature.
>
> Thanks,
> Zike Yang
>
>
> Zike Yang
>
> On Tue, Feb 28, 2023 at 10:25 AM Jiaqi Shen <gl...@gmail.com>
> wrote:
> >
> > Context.
> > - PIP-74
> >
> https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits
> > - PR-8965 https://github.com/apache/pulsar/pull/8965
> > - PR-15216 https://github.com/apache/pulsar/pull/15216
> >
> > Hello, community:
> >
> > There are some questions about PIP-74 I want to figure out.
> >
> > PIP-74 and its implementation specifie how to limit *client* memory. But
> in
> > our scenario, the *client* is usually reused. It is more intuitive to
> limit
> > a single *producer* or *consumer*. So why not let the producer/consumer
> > have their own MemoryLimitController? And shoud we split the
> > memoryLimit setting
> > into consumerMemoryLimit and producerMemoryLimit?  Are there any other
> > considerations to limit *client* memory?
> >
> > If you know why it needs to be designed like this, please leave your
> > comment. Thanks!
> >
> > Thanks,
> > Jiaqi Shen
>

Re: [DISCUSS] Why not split `memoryLimit` into `consumerMemoryLimit ` and `producerMemoryLimit `

[Zike Yang <zi...@apache.org>]

Hi, JIaqi

Thanks for initiating this discussion.

Do you have any cases that need separate limit control for producers/consumers?

We have max pending queue size and max message size for the producer,
and receiver queue size for the consumer. Although they hardly control
the limit as the byte-granular, they seem to be sufficient for the
current needs.

If there is a strong case, I think we can consider this feature.

Thanks,
Zike Yang


Zike Yang

On Tue, Feb 28, 2023 at 10:25 AM Jiaqi Shen <gl...@gmail.com> wrote:
>
> Context.
> - PIP-74
> https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits
> - PR-8965 https://github.com/apache/pulsar/pull/8965
> - PR-15216 https://github.com/apache/pulsar/pull/15216
>
> Hello, community:
>
> There are some questions about PIP-74 I want to figure out.
>
> PIP-74 and its implementation specifie how to limit *client* memory. But in
> our scenario, the *client* is usually reused. It is more intuitive to limit
> a single *producer* or *consumer*. So why not let the producer/consumer
> have their own MemoryLimitController? And shoud we split the
> memoryLimit setting
> into consumerMemoryLimit and producerMemoryLimit?  Are there any other
> considerations to limit *client* memory?
>
> If you know why it needs to be designed like this, please leave your
> comment. Thanks!
>
> Thanks,
> Jiaqi Shen

Re: [DISCUSS] Why not split `memoryLimit` into `consumerMemoryLimit ` and `producerMemoryLimit `

[Haiting Jiang <ji...@gmail.com>]

Hi Jiaqi,

From what I see, this memory limitation is used for protecting the
client from OOM when processing too many messages in the client. So
setting a single client limitation is more easy for users and producer
and consumer can share the memory.

Normally, we should set the limitation number to just small enough not
causing OOM, but not just large enough for normal message processing
which is hard to calculate. And having separate limitations for
producer and consumer may make the choice of limitation number more
tricky.

Thanks,
Haiting

On Tue, Feb 28, 2023 at 10:26 AM Jiaqi Shen <gl...@gmail.com> wrote:
>
> Context.
> - PIP-74
> https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits
> - PR-8965 https://github.com/apache/pulsar/pull/8965
> - PR-15216 https://github.com/apache/pulsar/pull/15216
>
> Hello, community:
>
> There are some questions about PIP-74 I want to figure out.
>
> PIP-74 and its implementation specifie how to limit *client* memory. But in
> our scenario, the *client* is usually reused. It is more intuitive to limit
> a single *producer* or *consumer*. So why not let the producer/consumer
> have their own MemoryLimitController? And shoud we split the
> memoryLimit setting
> into consumerMemoryLimit and producerMemoryLimit?  Are there any other
> considerations to limit *client* memory?
>
> If you know why it needs to be designed like this, please leave your
> comment. Thanks!
>
> Thanks,
> Jiaqi Shen