Console Consumer - Multiple Issues when Consuming from a topic and resuming from last committed offset.

["M. Manna" <ma...@gmail.com>]

I tried this numerous times (regardless of PLAINTEXT/SSL connnections). The
setup is to do with a single node (1 Zeek, 1 Broker) as mentioned in the
startup docs on Kafka site.

1) Follow kafka documentation to initialise all Zk and Kafka props.
2) Enable automatic topic creation with a minimum partition 1 in broker
properties.
3) Start Zk and Kafka.
4) Use the standard shipped producer.properties file with
bootstrap.servers, key.serializer, and value.serializer properties values.
5) Use the standard shipped consumer.properties file with the following
settings:

max.poll.interval.ms=21414111
#consumer group id
group.id=group1
key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
enable.auto.commit=true
auto.offset.reset=earliest
fetch.min.bytes=1
session.timeout.ms=20000

Since we are using console consumer, I kept the enable.auto.commit=true (i
think every 5 secs is the default interval for committing auto)

4) Run following command:

kafka-console-producer.bat --topic test --broker-list localhost:9092
--producer.config ..\..\config\producer.properties
> HELLO KAFKA!!!
> FINISHING
> ^ C (keyboard close producer)

5) Run the following command
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test
--new-consumer --consumer.config ..\..\config\consumer.properties
--from-beginning
HELLO KAFKA!!!
FINISHING
(wait for 5 seconds or more)
^C (keyboard close consumer)

6) Now repeat #4 with some more messages and close producer
7) Repeat #5.- confirm that all messages (old + new ones) are being
displayed

From the help documentation for console-consumer.

> --from-beginning      If the conumer doesn not already have an established
> offset to consume from, start from the earliest message present in the log
> rather than the latest message.


The consumer group is present so any offset committed correctly should have
been used by a newly added consumer to retrieve offset details.
`auto.offset.reset=earliest` but that's not relevant since offsets have
been committed and should have been found.

When I did the following:
kafka-consumer-groups.bat --bootstrap.server localhost:9092 --describe
--group group1 --new-consumer

I see "Error group1 does not exist".

I also tried to use kafka-consumer-offset-checker - but ran into the error
" org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /consumers/group1/offsets/test/0"

Which seems to be a bug as mentioned by -
https://stackoverflow.com/questions/22563401/rebalancing-issue-while-reading-messages-in-kafka


My question is - why does the console consumer fail to retrieve the offset
information and loses data? Or is this expected? I didn't have any consumer
offset expiry default configs altered. So any info. is helpful.

I don't believe this is relevant to Windows/Linux but if anyone has this
example (with the same setup) working in Linux ONLY - it's good to know. I
can confirm that I can see all the consumer offsets created in the specific
log directory in broker prop file.

Kindest Regards,