You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by SenthilKumar K <se...@gmail.com> on 2020/08/21 09:51:41 UTC
Kafka Producer - Producing to Multiple Topics
Hi Team, We have deployed 150 node Kafka cluster on production for our use
case. Recently I have seen issue(s) in Kafka Producer Client.
Use Case:
<Incoming-Topic> --> (Consume)Stream App (Multiple Topologies) (Transform)
--> Kafka Producer Topology ( Produce it to Multiple Topics)
Initially, the data is written to a common buffer topic. The stream
topologies will consume from buffer topic and transform from one form to
JSON. Finally transformed output goes to Kafka Producer Topology. The
nature of the Kafka Producer Topology is to get input from Transform
Topology and Write it to stream specific topics.
No of Transformation Topologies: 200
No Of Kafka Producer Topologies: 100
Each Kafka Producer Topology will be producing on multiple topics( approx:
2000). Initially, it used to produce to 1000 topics, and now the number of
topics has increased hence the problem observed.
What is the recommended approach to produce data on multiple topics from
multiple clients?
Looking forward to your inputs to optimize the Kafka producer topology.
Thanks in advance!
Client Version:
<kafka-client.version>2.3.0</kafka-client.version>
Kafka Producer Configuration:
retries=3
linger.ms=5
buffer.memory=67108864
batch.size=32768
compression.type=snappy
--Senthil
Re: Kafka Producer - Producing to Multiple Topics
Posted by SenthilKumar K <se...@gmail.com>.
it would be great if someone provides input(s)/hint :) thanks!
--Senthil
On Fri, Aug 21, 2020 at 3:28 PM SenthilKumar K <se...@gmail.com>
wrote:
> Updating the Kafka broker version:
>
> Kafka Version: 2.4.1
>
> On Fri, Aug 21, 2020 at 3:21 PM SenthilKumar K <se...@gmail.com>
> wrote:
>
>> Hi Team, We have deployed 150 node Kafka cluster on production for our
>> use case. Recently I have seen issue(s) in Kafka Producer Client.
>>
>> Use Case:
>> <Incoming-Topic> --> (Consume)Stream App (Multiple Topologies)
>> (Transform) --> Kafka Producer Topology ( Produce it to Multiple Topics)
>>
>> Initially, the data is written to a common buffer topic. The stream
>> topologies will consume from buffer topic and transform from one form to
>> JSON. Finally transformed output goes to Kafka Producer Topology. The
>> nature of the Kafka Producer Topology is to get input from Transform
>> Topology and Write it to stream specific topics.
>>
>> No of Transformation Topologies: 200
>> No Of Kafka Producer Topologies: 100
>>
>> Each Kafka Producer Topology will be producing on multiple topics(
>> approx: 2000). Initially, it used to produce to 1000 topics, and now the
>> number of topics has increased hence the problem observed.
>>
>> What is the recommended approach to produce data on multiple topics from
>> multiple clients?
>>
>> Looking forward to your inputs to optimize the Kafka producer topology.
>> Thanks in advance!
>>
>> Client Version:
>>
>> <kafka-client.version>2.3.0</kafka-client.version>
>>
>>
>> Kafka Producer Configuration:
>>
>>
>> retries=3
>> linger.ms=5
>> buffer.memory=67108864
>> batch.size=32768
>> compression.type=snappy
>>
>>
>> --Senthil
>>
>
Re: Kafka Producer - Producing to Multiple Topics
Posted by SenthilKumar K <se...@gmail.com>.
it would be great if someone provides input(s)/hint :) thanks!
--Senthil
On Fri, Aug 21, 2020 at 3:28 PM SenthilKumar K <se...@gmail.com>
wrote:
> Updating the Kafka broker version:
>
> Kafka Version: 2.4.1
>
> On Fri, Aug 21, 2020 at 3:21 PM SenthilKumar K <se...@gmail.com>
> wrote:
>
>> Hi Team, We have deployed 150 node Kafka cluster on production for our
>> use case. Recently I have seen issue(s) in Kafka Producer Client.
>>
>> Use Case:
>> <Incoming-Topic> --> (Consume)Stream App (Multiple Topologies)
>> (Transform) --> Kafka Producer Topology ( Produce it to Multiple Topics)
>>
>> Initially, the data is written to a common buffer topic. The stream
>> topologies will consume from buffer topic and transform from one form to
>> JSON. Finally transformed output goes to Kafka Producer Topology. The
>> nature of the Kafka Producer Topology is to get input from Transform
>> Topology and Write it to stream specific topics.
>>
>> No of Transformation Topologies: 200
>> No Of Kafka Producer Topologies: 100
>>
>> Each Kafka Producer Topology will be producing on multiple topics(
>> approx: 2000). Initially, it used to produce to 1000 topics, and now the
>> number of topics has increased hence the problem observed.
>>
>> What is the recommended approach to produce data on multiple topics from
>> multiple clients?
>>
>> Looking forward to your inputs to optimize the Kafka producer topology.
>> Thanks in advance!
>>
>> Client Version:
>>
>> <kafka-client.version>2.3.0</kafka-client.version>
>>
>>
>> Kafka Producer Configuration:
>>
>>
>> retries=3
>> linger.ms=5
>> buffer.memory=67108864
>> batch.size=32768
>> compression.type=snappy
>>
>>
>> --Senthil
>>
>
Re: Kafka Producer - Producing to Multiple Topics
Posted by SenthilKumar K <se...@gmail.com>.
Updating the Kafka broker version:
Kafka Version: 2.4.1
On Fri, Aug 21, 2020 at 3:21 PM SenthilKumar K <se...@gmail.com>
wrote:
> Hi Team, We have deployed 150 node Kafka cluster on production for our
> use case. Recently I have seen issue(s) in Kafka Producer Client.
>
> Use Case:
> <Incoming-Topic> --> (Consume)Stream App (Multiple Topologies) (Transform)
> --> Kafka Producer Topology ( Produce it to Multiple Topics)
>
> Initially, the data is written to a common buffer topic. The stream
> topologies will consume from buffer topic and transform from one form to
> JSON. Finally transformed output goes to Kafka Producer Topology. The
> nature of the Kafka Producer Topology is to get input from Transform
> Topology and Write it to stream specific topics.
>
> No of Transformation Topologies: 200
> No Of Kafka Producer Topologies: 100
>
> Each Kafka Producer Topology will be producing on multiple topics( approx:
> 2000). Initially, it used to produce to 1000 topics, and now the number of
> topics has increased hence the problem observed.
>
> What is the recommended approach to produce data on multiple topics from
> multiple clients?
>
> Looking forward to your inputs to optimize the Kafka producer topology.
> Thanks in advance!
>
> Client Version:
>
> <kafka-client.version>2.3.0</kafka-client.version>
>
>
> Kafka Producer Configuration:
>
>
> retries=3
> linger.ms=5
> buffer.memory=67108864
> batch.size=32768
> compression.type=snappy
>
>
> --Senthil
>