You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Xiong, Pin" <pi...@foxmail.com> on 2022/03/31 14:14:58 UTC

[Question] What is the best practice in consuming multiple topics?

To whom it may concern,

I have a very different problem when using Kafka, and hope someone can help me!

Background

I need to consume several different topics in my java project. For each topic, they has different configurations, such as max.pull.records, but the same consumer group.

Also, there are different business logic for each topic.


How did I do

I have created different KafkaConsumer instances with different topics. Because I can setup the configurations only when creating the KafkaConsumer instance.


Problem

Some topic partitions cannot be consumed. This is really a mazing phenomenon! I found that the client.id will be changed when I redeployed and restarted the project.

All of KafkaConsumer share with the global static variable CONSUMER_CLIENT_ID_SEQUENCE in ConsumerConfig if I didn’t set client.id .

Is it the best practice to configure the different client.id for each KafkaConsumer instance? Or who can tell me how to solve my issue?


Looking forward to your reply!

Thanks，
Pin

Re: [Question] What is the best practice in consuming multiple topics?

Posted by "Xiong, Pin" <pi...@foxmail.com>.
Hi BaiYang,

Thanks for your reply!

For your question, my answer is below:

I don’t need to care about Kafka brokers because I use AWS’s MSK product. For producers, I didn’t restart it, which is a cluster services and provided by other team.Consumers will be restarted when my application is redeployed.


Thanks，
Pin

> On Apr 1, 2022, at 23:10, BaiYang <by...@163.com> wrote:
> 
> how about stop the kafka brokers and restart them and restart producers and consumers?
> 
> 
> 
> 
> ---- 回复的原邮件 ----
> | 发件人 | Xiong, Pin<pi...@foxmail.com> |
> | 日期 | 2022年03月31日 22:14 |
> | 收件人 | users@kafka.apache.org<us...@kafka.apache.org> |
> | 抄送至 | |
> | 主题 | [Question] What is the best practice in consuming multiple topics? |
> To whom it may concern,
> 
> I have a very different problem when using Kafka, and hope someone can help me!
> 
> Background
> 
> I need to consume several different topics in my java project. For each topic, they has different configurations, such as max.pull.records, but the same consumer group.
> 
> Also, there are different business logic for each topic.
> 
> 
> How did I do
> 
> I have created different KafkaConsumer instances with different topics. Because I can setup the configurations only when creating the KafkaConsumer instance.
> 
> 
> Problem
> 
> Some topic partitions cannot be consumed. This is really a mazing phenomenon! I found that the client.id will be changed when I redeployed and restarted the project.
> 
> All of KafkaConsumer share with the global static variable CONSUMER_CLIENT_ID_SEQUENCE in ConsumerConfig if I didn’t set client.id .
> 
> Is it the best practice to configure the different client.id for each KafkaConsumer instance? Or who can tell me how to solve my issue?
> 
> 
> Looking forward to your reply!
> 
> Thanks，
> Pin
>

Re: [Question] What is the best practice in consuming multiple topics?

Posted by BaiYang <by...@163.com>.
how about stop the kafka brokers and restart them and restart producers and consumers?




---- 回复的原邮件 ----
| 发件人 | Xiong, Pin<pi...@foxmail.com> |
| 日期 | 2022年03月31日 22:14 |
| 收件人 | users@kafka.apache.org<us...@kafka.apache.org> |
| 抄送至 | |
| 主题 | [Question] What is the best practice in consuming multiple topics? |
To whom it may concern,

I have a very different problem when using Kafka, and hope someone can help me!

Background

I need to consume several different topics in my java project. For each topic, they has different configurations, such as max.pull.records, but the same consumer group.

Also, there are different business logic for each topic.


How did I do

I have created different KafkaConsumer instances with different topics. Because I can setup the configurations only when creating the KafkaConsumer instance.


Problem

Some topic partitions cannot be consumed. This is really a mazing phenomenon! I found that the client.id will be changed when I redeployed and restarted the project.

All of KafkaConsumer share with the global static variable CONSUMER_CLIENT_ID_SEQUENCE in ConsumerConfig if I didn’t set client.id .

Is it the best practice to configure the different client.id for each KafkaConsumer instance? Or who can tell me how to solve my issue?


Looking forward to your reply!

Thanks，
Pin