Apache Kafka consumer consumes messages with "partition" option, but not with "group" option

["Geithner, Wolfgang Dr." <W....@gsi.de>]

This is a copy of a topic I posted in stackoverflow (https://stackoverflow.com/questions/76458064/apache-kafka-consumer-consumes-messages-with-partition-option-but-not-with-g), where I didn't get any answer yet. Searching the web did not yield any helpful reults either. Hence, I am addressing to this mailing list:

I am running a plain Apache Kafka server (version 3.4.1) which I would like to connect to a Telegraf consumer. The Telegraf ```[[inputs.kafka_consumer]]``` plugin has the option to consume by Kafka "group". When staring Telegraf, I get an error message

    [inputs.kafka_consumer] Error in plugin: consume: kafka server: Request was for a consumer group that is not coordinated by this broker

Hence, I started to investigate my setup by using the Kafka console tools and found that when executing

     ./kafka-console-consumer.sh --bootstrap-server myserver:9092 --topic test --partition 0

and sending messages via ```kafka-console-consumer.sh```, these messages pop up in the console "consumer" window as expected.

In contrast to this, when I run

     ./kafka-console-consumer.sh --bootstrap-server myserver:9092 --topic test --group my-group

nothing happens in the "consumer" window. Furthermore, the command

    ./kafka-consumer-groups.sh --bootstrap-server myserver:9092 --list

yields nothing.

What do I have to do to cause the consumer with the "group" option to "see" the messages produced to the topic "test"? Ultimately, how can I solve the Telegraf error?

Re: Apache Kafka consumer consumes messages with "partition" option, but not with "group" option

[Neeraj Vaidya <ne...@yahoo.co.in.INVALID>]

 Hi,
If you have a large number of partitions in your topic, it can take a really long while before you start seeing messages on the console.

So, using the partition id is the right approach. But just need to be patient at the command-line. Out of interest, how long did you wait for the output from console consumer ?

If you need to know the partition id, you will need to use a custom program to compute it based on the key. (You could have a look at the murmur2 source code on the Kafka github repository and try to create a simple command line tool to compute the partition id using the key).

However, using --group option will only set the consumer group id of your instance of the kafka-console-consumer.sh.

Regards,
Neeraj
     On Tuesday, 13 June, 2023 at 05:26:24 pm GMT+10, Geithner, Wolfgang Dr. <w....@gsi.de> wrote:  
 
 This is a copy of a topic I posted in stackoverflow (https://stackoverflow.com/questions/76458064/apache-kafka-consumer-consumes-messages-with-partition-option-but-not-with-g), where I didn't get any answer yet. Searching the web did not yield any helpful reults either. Hence, I am addressing to this mailing list:

I am running a plain Apache Kafka server (version 3.4.1) which I would like to connect to a Telegraf consumer. The Telegraf ```[[inputs.kafka_consumer]]``` plugin has the option to consume by Kafka "group". When staring Telegraf, I get an error message

    [inputs.kafka_consumer] Error in plugin: consume: kafka server: Request was for a consumer group that is not coordinated by this broker

Hence, I started to investigate my setup by using the Kafka console tools and found that when executing

    ./kafka-console-consumer.sh --bootstrap-server myserver:9092 --topic test --partition 0

and sending messages via ```kafka-console-consumer.sh```, these messages pop up in the console "consumer" window as expected.

In contrast to this, when I run

    ./kafka-console-consumer.sh --bootstrap-server myserver:9092 --topic test --group my-group

nothing happens in the "consumer" window. Furthermore, the command

    ./kafka-consumer-groups.sh --bootstrap-server myserver:9092 --list

yields nothing.

What do I have to do to cause the consumer with the "group" option to "see" the messages produced to the topic "test"? Ultimately, how can I solve the Telegraf error?