You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Rajini Sivaram (JIRA)" <ji...@apache.org> on 2017/07/05 08:42:02 UTC
[jira] [Commented] (KAFKA-5537) Subscribe Earliest is not working
as in 0.10.2.1
[ https://issues.apache.org/jira/browse/KAFKA-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074428#comment-16074428 ]
Rajini Sivaram commented on KAFKA-5537:
---------------------------------------
[~michael.andre.pearce] I have run your test and it passes consistently for me if I increase the first sleep (before produce) from 4000 to 8000. Your logs for 0.11.0 show that you are producing before partitions are assigned. With offsets reset to "latest", this will mean that the messages produced before assignment are not consumed.
From your logs for 0.11.0:
{quote}
2017-06-30 23:02:15 INFO AbstractCoordinator:597 - Discovered coordinator localhost:52196 (id: 2147483646 rack: null) for group test-group.
017-06-30 23:02:15 INFO AbstractCoordinator:597 - Discovered coordinator localhost:52196 (id: 2147483646 rack: null) for group test-group.
...
017-06-30 23:02:18 TRACE KafkaProducer:740 - Sending record ProducerRecord(topic=topic, partition=null, headers=RecordHeaders(headers = [], isReadOnly = true), key=null, value=[B@62379589, timestamp=null) with callback null to topic topic partition 0
...
017-06-30 23:02:21 INFO ConsumerCoordinator:262 - Setting newly assigned partitions [] for group test-group
2017-06-30 23:02:21 INFO ConsumerCoordinator:262 - Setting newly assigned partitions (topic-0) for group test-group
{quote}
Messages are produced 4 seconds after the consumer was created. But at this point, rebalancing hasn't completed and no partitions have been assigned to the consumers. When the partitions are assigned 3 seconds later, consumers start consuming from the "latest" offset at point, ignoring already produced messages. With {{AUTO_OFFSET_RESET_CONFIG=earliest}}, this would have worked.
With lower {{group.initial.rebalance.delay.ms}}, the test works since rebalance completes sooner. When your test was waiting for 60 seconds, can you check if the wait was before producing messages when offset reset strategy is {{latest}}?
> Subscribe Earliest is not working as in 0.10.2.1
> ------------------------------------------------
>
> Key: KAFKA-5537
> URL: https://issues.apache.org/jira/browse/KAFKA-5537
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.11.0.0
> Reporter: Michael Andre Pearce
> Priority: Critical
> Attachments: kafka_0.10.2.1.log, kafka_0.11.0.0.log, KafkaSub.java, KafkaSubLatest.java
>
>
> We have seen issue with subscription where auto offset when set to earliest (and also latest) does not behave the same as in 0.10.2.1 release.
> We have managed to create a repeatable test for this, which passes when pointing to 0.10.2.1 broker.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)