How many producers can publish to same topic concurrently?

[Pulkit Manchanda <pu...@gmail.com>]

Hi All,

I am working on a use case where my multiple producers will be publishing
to the same topic.
For this I need to know what is the MAX_LIMIT of same producers writing to
the same topic?

Also is there any case that the create a pool of producers and these
instances can be shared?

Thanks
Pulkit

RE: How many producers can publish to same topic concurrently?

["Opara, P. (Piotr)" <PI...@ing.com.INVALID>]

Hi Koushik

Will you share the information about hardware you have?

Pozdrawiam / Kind Regards

Piotr Opara
DevOps Engineer

-----Original Message-----
From: Koushik Chitta [mailto:kchitta@microsoft.com.INVALID] 
Sent: 26 lipca 2018 20:58
To: users@kafka.apache.org
Subject: RE: How many producers can publish to same topic concurrently?

We had ~75000 producers writing to a single topic  on 10 broker cluster in Windows environment, yet to reach the max. limit considering the system resources used. I guess our bottleneck might be the no. of tcp connections to a broker.


-----Original Message-----
From: Pulkit Manchanda <pu...@gmail.com> 
Sent: Tuesday, July 24, 2018 6:16 AM
To: users@kafka.apache.org
Subject: How many producers can publish to same topic concurrently?

Hi All,

I am working on a use case where my multiple producers will be publishing to the same topic.
For this I need to know what is the MAX_LIMIT of same producers writing to the same topic?

Also is there any case that the create a pool of producers and these instances can be shared?

Thanks
Pulkit

ING Business Shared Services B.V. z siedzibą w Amsterdamie, Holandia, VAT PL 526-319-58-54, działająca w Polsce w formie oddziału, pod firmą ING Business Shared Services B.V. spółka z ograniczoną odpowiedzialnością Oddział w Polsce z siedzibą w Katowicach, ul. Konduktorska 35, 40-155 Katowice, NIP: 2050005130, wpisana do rejestru przedsiębiorców Krajowego Rejestru Sądowego prowadzonego przez Sąd Rejonowy Katowice-Wschód w Katowicach, VIII Wydział Gospodarczy Krajowego Rejestru Sądowego pod numerem KRS 0000702305.

Re: How many producers can publish to same topic concurrently?

[Pulkit Manchanda <pu...@gmail.com>]

Thanks Koushik,
It will be helpful if you can share the h/w resources are you using.

Pulkit

On Thu, Jul 26, 2018 at 2:57 PM, Koushik Chitta <
kchitta@microsoft.com.invalid> wrote:

> We had ~75000 producers writing to a single topic  on 10 broker cluster in
> Windows environment, yet to reach the max. limit considering the system
> resources used. I guess our bottleneck might be the no. of tcp connections
> to a broker.
>
>
> -----Original Message-----
> From: Pulkit Manchanda <pu...@gmail.com>
> Sent: Tuesday, July 24, 2018 6:16 AM
> To: users@kafka.apache.org
> Subject: How many producers can publish to same topic concurrently?
>
> Hi All,
>
> I am working on a use case where my multiple producers will be publishing
> to the same topic.
> For this I need to know what is the MAX_LIMIT of same producers writing to
> the same topic?
>
> Also is there any case that the create a pool of producers and these
> instances can be shared?
>
> Thanks
> Pulkit
>

RE: How many producers can publish to same topic concurrently?

[Koushik Chitta <kc...@microsoft.com.INVALID>]

We had ~75000 producers writing to a single topic  on 10 broker cluster in Windows environment, yet to reach the max. limit considering the system resources used. I guess our bottleneck might be the no. of tcp connections to a broker.


-----Original Message-----
From: Pulkit Manchanda <pu...@gmail.com> 
Sent: Tuesday, July 24, 2018 6:16 AM
To: users@kafka.apache.org
Subject: How many producers can publish to same topic concurrently?

Hi All,

I am working on a use case where my multiple producers will be publishing to the same topic.
For this I need to know what is the MAX_LIMIT of same producers writing to the same topic?

Also is there any case that the create a pool of producers and these instances can be shared?

Thanks
Pulkit