You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "sanghyeok An (Jira)" <ji...@apache.org> on 2024/05/06 02:13:00 UTC
[jira] [Created] (KAFKA-16670) KIP-848 : Consumer will not receive assignment forever because of concurrent issue.
sanghyeok An created KAFKA-16670:
------------------------------------
Summary: KIP-848 : Consumer will not receive assignment forever because of concurrent issue.
Key: KAFKA-16670
URL: https://issues.apache.org/jira/browse/KAFKA-16670
Project: Kafka
Issue Type: Bug
Reporter: sanghyeok An
*Related Code*
* Consumer get assignment Successfully :
** [https://github.com/chickenchickenlove/new-consumer-error/blob/8c1d74db1ec60350c28f5ed25f595559180dc603/src/test/java/com/example/MyTest.java#L35-L57]
* Consumer get be stuck Forever because of concurrent issue:
** https://github.com/chickenchickenlove/new-consumer-error/blob/8c1d74db1ec60350c28f5ed25f595559180dc603/src/test/java/com/example/MyTest.java#L61-L79
*Unexpected behaviour*
[|https://github.com/chickenchickenlove/new-consumer-error#unexpected-behaviour] * Broker is sufficiently slow.
* When a KafkaConsumer is created and immediately subscribes to a topic
If both conditions are met, {{Consumer}} can potentially never receive {{TopicPartition}} assignments and become stuck indefinitely.
In case of new broker and new consumer, when consumer are created, consumer background thread send a request to broker. (I guess groupCoordinator Heartbeat request). In that time, if broker does not load metadata from {{{}__consumer_offset{}}}, broker will start to schedule load metadata. After broker load metadata completely, consumer background thread think 'this broker is valid group coordinator'.
However, consumer can send {{subscribe}} request to broker before {{broker}} reply about {{{}groupCoordinator HeartBeat Request{}}}. In that case, consumer seems to be stuck.
If both conditions are met, the {{Consumer}} can potentially never receive {{TopicPartition}} assignments and may become indefinitely stuck. In the case of a new {{broker}} and new {{{}consumer{}}}, when the consumer is created, {{consumer background thread}} start to send a request to the broker. (I believe this is a {{{}GroupCoordinator Heartbeat request{}}}) During this time, if the {{broker}} has not yet loaded metadata from {{{}__consumer_offsets{}}}, it will begin to schedule metadata loading. Once the broker has completely loaded the metadata, the {{consumer background thread}} recognizes this broker as a valid group coordinator. However, there is a possibility that the {{consumer}} can send a {{subscribe request}} to the {{broker}} before the {{broker}} has replied to the {{{}GroupCoordinator Heartbeat Request{}}}. In such a scenario, the {{consumer}} appears to be stuck.
You can check this scenario, in the {{{}src/test/java/com/example/MyTest#should_fail_because_consumer_try_to_poll_before_background_thread_get_valid_coordinator{}}}. If there is no sleep time to wait {{{}GroupCoordinator Heartbeat Request{}}}, {{consumer}} will be always stuck. If there is a little sleep time, {{consumer}} will always receive assignment.
README : https://github.com/chickenchickenlove/new-consumer-error/blob/main/README.md
--
This message was sent by Atlassian Jira
(v8.20.10#820010)