You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Jan Burkhardt (JIRA)" <ji...@apache.org> on 2017/07/17 11:43:00 UTC
[jira] [Updated] (KAFKA-5600) GroupMetadataManager doesn't read
Offsets of segmented log correctly
[ https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan Burkhardt updated KAFKA-5600:
---------------------------------
Description:
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and needs to reload offsets it starts at a wrong position
Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries one by one and then closes the consumer. This leads to a 2nd segment in __consumer_offsets-27
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server localhost:9092 --topic test --consumer-property group.id=testgroup
{code}
Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end
The Reason for this behavior is the closing brace of the while loop in GroupMetadataManager#loadGroupsAndOffsets at a wrong position
was:
After long investigation we found a Problem in Kafka.
When a __consumer_offsets partition gets segmented and Kafka is restarted and needs to reload offsets it starts at a wrong position
Reproduction:
1.) Start zookeeper and kafka as is from the archive
{code}
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh config/zookeeper.properties
KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh config/server.properties
{code}
2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries one by one and then closes the consumer. This leads to a 2nd segment in __consumer_offsets-27
4.) Stop and restart the Kafka broker
5.) Start any consumer on topic test and group testgroup
{code}
bin/kafka-console-consumer.sh --from-beginning --bootstrap-server localhost:9092 --topic test --consumer-property group.id=testgroup
{code}
Is:
the consumer starts at the segmentation boundary
Expected:
the consumer starts at the end
The Reason for this behavior is the closing brace of the while loop in GroupMetadataManager#loadGroupsAndOffsets
> GroupMetadataManager doesn't read Offsets of segmented log correctly
> --------------------------------------------------------------------
>
> Key: KAFKA-5600
> URL: https://issues.apache.org/jira/browse/KAFKA-5600
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.10.2.0, 0.11.0.0
> Environment: any
> Reporter: Jan Burkhardt
> Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and needs to reload offsets it starts at a wrong position
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries one by one and then closes the consumer. This leads to a 2nd segment in __consumer_offsets-27
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in GroupMetadataManager#loadGroupsAndOffsets at a wrong position
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)