You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Xin (Jira)" <ji...@apache.org> on 2023/12/29 03:05:00 UTC
[jira] [Commented] (KAFKA-16050) consumer was removed from group，but still can poll data from kafka, data duplicate
[ https://issues.apache.org/jira/browse/KAFKA-16050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801087#comment-17801087 ]
Xin commented on KAFKA-16050:
-----------------------------
Finally Ifound the rreson
consumer Rejoin operation was delay for a long time, and another consumer has been assigned the partitions
Can we optimize these process？
> consumer was removed from group，but still can poll data from kafka, data duplicate
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-16050
> URL: https://issues.apache.org/jira/browse/KAFKA-16050
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.3.1
> Reporter: Xin
> Priority: Major
>
> I have 3 brokers: b1,b2,b3
> a topic : test, partitiion5,replication3
> 3 consumer in 1 group: consumer1,consumer2,consumer3
> groupid: xx
>
> consumer1 running in b1
> consumer2 running in b2
> consumer3 running in b3
> ./kafka-console-consumer.sh --bootstrap-server localhost:9093 --group xx --topic test --from-beginning
>
> b2's clock changed, consumer2 was removed from group xx(reason: removing member consumer-2 on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)
> kafka-consumer-groups.sh can't see any record about cosumer2
> ./kafka-consumer-groups.sh --bootstrap-server localhost:9093 --all-topics --describe --all-groups
>
> Then consumer rebalanced, partiitons assigned to consumer2 was assigned to other consumer
> Although consumer2 was removed from group xx ,BUT still poll data from kafka ,kafka can't find it
> After rebalance another consumer poll the same partition with consumer2
> This make data was poll duplicate
>
>
>
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)