Design Question About Client Limits

[Dave Fisher <da...@comcast.net>]

Hi -

I am doing some design work for a message based workflow service I am thinking about building.

What is the limit on the number of simultaneously open topic producers on a client node?

Regards,
Dave

Re: Design Question About Client Limits

[Dave Fisher <da...@comcast.net>]

Hi Andrews and Matteo,

Thanks guys. I’ll treat the issue as a second phase design concern where I can time out resources associated with unused producers in the client.

Regards,
Dave

> On Nov 1, 2017, at 2:39 PM, Sahaya Andrews <an...@apache.org> wrote:
> 
> We have tested 50,000 topics from a client(java) box, each topic with
> one producer and one consumer. On an average around, 80-100 topics
> loaded per second. On the server side, it had 3 brokers and 3 bookies.
> 
> Andrews.
> 
> On Wed, Nov 1, 2017 at 1:28 PM, Dave Fisher <da...@comcast.net> wrote:
>> Hi -
>> 
>> I am doing some design work for a message based workflow service I am thinking about building.
>> 
>> What is the limit on the number of simultaneously open topic producers on a client node?
>> 
>> Regards,
>> Dave

Re: Design Question About Client Limits

[Sahaya Andrews <an...@apache.org>]

We have tested 50,000 topics from a client(java) box, each topic with
one producer and one consumer. On an average around, 80-100 topics
loaded per second. On the server side, it had 3 brokers and 3 bookies.

Andrews.

On Wed, Nov 1, 2017 at 1:28 PM, Dave Fisher <da...@comcast.net> wrote:
> Hi -
>
> I am doing some design work for a message based workflow service I am thinking about building.
>
> What is the limit on the number of simultaneously open topic producers on a client node?
>
> Regards,
> Dave

Re: Design Question About Client Limits

[Matteo Merli <ma...@gmail.com>]

Hi Dave,

I don't think there's a particular limit to it. For a single application,
the idea is to share a single Pulsar client instance.
That makes sure to reuse a lot of resources (threads, TCP connections,
etc..)

For each producer, the biggest amount of memory would be related to the
queue of outstanding messages. The default for the queue size is 1000, but
it could be reduced in case the required throughput for each producer is is
not that high.

I haven't done the real check on the amount of memory (I don't expect it to
be super-huge), though it could be tested using the `pulsar-perf` producer
from CLI. The tool can easily simulate N producers on N topics.

Matteo

On Wed, Nov 1, 2017 at 1:28 PM Dave Fisher <da...@comcast.net> wrote:

> Hi -
>
> I am doing some design work for a message based workflow service I am
> thinking about building.
>
> What is the limit on the number of simultaneously open topic producers on
> a client node?
>
> Regards,
> Dave
>
-- 
Matteo Merli
<mm...@apache.org>