[jira] [Commented] (KAFKA-5600) GroupMetadataManager doesn't read offsets of segmented logs correctly

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089744#comment-16089744 ] 

ASF GitHub Bot commented on KAFKA-5600:
---------------------------------------

GitHub user bjrke opened a pull request:

    https://github.com/apache/kafka/pull/3538

    KAFKA-5600: fix GroupMetadataManager doesn't read offsets of segmente…

    …d log correctly
    
    the while loop was too big and need to be closed earlier
    to see the fix, ignore whitespace since most of it is indentation
    
    this bug was introduced by commit
    5bd06f1d542e6b588a1d402d059bc24690017d32

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/justsocialapps/kafka trunk

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/3538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3538
    
----
commit 6b625780dd67c1dc04a90c58f99f77efd18b9d29
Author: Jan Burkhardt <ja...@just.social>
Date:   2017-07-14T14:30:36Z

    KAFKA-5600: fix GroupMetadataManager doesn't read offsets of segmented log correctly
    
    the while loop was too big and need to be closed earlier
    to see the fix, ignore whitespace since most of it is indentation
    
    this bug was introduced by commit
    5bd06f1d542e6b588a1d402d059bc24690017d32

----


> GroupMetadataManager doesn't read offsets of segmented logs correctly
> ---------------------------------------------------------------------
>
>                 Key: KAFKA-5600
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5600
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.10.2.0, 0.11.0.0
>         Environment: any
>            Reporter: Jan Burkhardt
>            Priority: Critical
>              Labels: reliability
>             Fix For: 0.11.0.1
>
>         Attachments: KafkaErrorConsumer.java, KafkaErrorProducer.java
>
>
> After long investigation we found a Problem in Kafka.
> When a __consumer_offsets partition gets segmented and Kafka is restarted and needs to reload offsets, consumers will start at a wrong position when metadata and offset events are in both segments.
> Reproduction:
> 1.) Start zookeeper and kafka as is from the archive
> {code}
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/zookeeper-server-start.sh config/zookeeper.properties
> KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" bin/kafka-server-start.sh config/server.properties
> {code}
> 2.) Start [^KafkaErrorProducer.java] which adds 1M log entries to the topic test
> 3.) Start [^KafkaErrorConsumer.java] which starts a consumer, reads 100 entries one by one and then closes the consumer. This leads to a 2nd segment in /tmp/kafka-logs/__consumer_offsets-27. This step takes some time (around 5mins). The close of the consumer is needed to have metadata events in the segments too.
> 4.) Stop and restart the Kafka broker
> 5.) Start any consumer on topic test and group testgroup
> {code}
> bin/kafka-console-consumer.sh --from-beginning --bootstrap-server localhost:9092 --topic test --consumer-property group.id=testgroup
> {code}
> Is:
> the consumer starts at the segmentation boundary
> Expected:
> the consumer starts at the end
> The Reason for this behavior is the closing brace of the while loop in GroupMetadataManager#loadGroupsAndOffsets at a wrong position introduced with commit https://github.com/apache/kafka/commit/5bd06f1d542e6b588a1d402d059bc24690017d32
> I will prepare a pull request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)