You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Gregory Koshelev (Jira)" <ji...@apache.org> on 2019/08/20 10:15:00 UTC
[jira] [Comment Edited] (KAFKA-8104) Consumer cannot rejoin to the
group after rebalancing
[ https://issues.apache.org/jira/browse/KAFKA-8104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911182#comment-16911182 ]
Gregory Koshelev edited comment on KAFKA-8104 at 8/20/19 10:14 AM:
-------------------------------------------------------------------
[~guozhang], I found server.log (there are 3 brokers, but only one of them has relevant log entries in log):
{code}
[2019-03-10 02:46:43,627] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10693 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:43,630] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10693 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:45,397] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10693 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-7-cc600ccb-72b6-4582-b288-3ea966f33c7c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-1-f284c5c8-22fc-40d5-aa30-46c2875463bd in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-5-e608adfc-e9ad-427e-99b5-99ff1538611d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-6-daa568cc-a5ee-49a9-aa7c-bd34fbb1da52 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-2-da70d65b-53e6-4467-b06d-63f02ff89eec in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-12-a0cb16ff-46eb-4945-a87a-244a20261af9 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-10-c10cb15d-edc3-4c6a-b21a-68e2b8298284 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-16-d80c8a16-be83-48f3-ac9d-08431b87c198 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-5-75899722-91a9-439a-a0c5-f23eb7c6c93f in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-11-14fe64e9-1553-49e1-abf9-5ea623847c84 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-3-9b62bde0-ff71-46f7-b6a4-7be14e6055b5 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-14-42336dad-be7c-49da-a76b-72d13a856943 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-6-1d57e91e-a6cd-4917-9560-19fa6579d02e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-13-a310873e-5c66-402c-a031-0b2fce8eb4cc in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-1-7c591ed8-01d8-4f2c-b756-a5c3d3f63eea in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-7-70c381bc-5d32-4e2c-923b-2c15c0f15aa2 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-15-5de2a785-0cd8-43f8-b3e2-6b1da2983910 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-10-92f6d7a8-cc6c-4091-9eb4-1a8be9a08541 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-5-bd971902-bea5-49ef-83f6-3788e7363924 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-11-18b7fff3-f477-44c8-bece-4bde079bb578 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-2-bb9ff074-6738-4092-a2c0-f3d49e5e7e2a in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-9-ef6b36b3-376f-4a04-a0cb-709c8a28712b in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-13-57c73f24-4e8d-407e-874b-ab95409f7b1c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-10-265e41ea-c8c3-47f2-96d2-33309f975f29 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-15-f3ea2d41-09df-454a-a13b-c2007e0bff32 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-11-dcbc5baf-b7e4-433f-9580-b5e846418748 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-14-3cfd686d-e590-4055-9c8c-70f673a0a624 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-8-43d4c7af-a8f5-4616-8346-33bed246a253 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-11-bf244902-001f-460c-8cf5-e9ac5e7cb4a6 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-10-74f336ca-6a0f-4603-b93a-fcee81953f9f in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-1-65fe1595-6bff-4506-8a9b-6f35f3663f48 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-16-470b7eb9-942d-48fa-b8c1-d128ae21900d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-2-d587e29f-4d7e-4234-a774-21813af82415 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-7-a1fbd1a5-a4a8-42a8-bf7e-d6742edb8003 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-16-47eaa6a5-1fe9-4216-8afa-aaa5a7684755 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-10-1cba9737-26c2-4879-82c7-d3ef6399a2b3 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-13-b4031cae-43c3-45a4-b258-b9cfe2645eb6 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-e23f1744-7781-4e09-b394-6e3a09a5ed6d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-4-59459730-4118-4379-a208-83bfba9a751e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-9-4291927f-119f-4610-bd00-f9503847ff05 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-6-b8d722e7-58d3-4e2f-af03-92ab01e23f8d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-14-b2504ede-535c-4884-aea5-03734661bb4d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-6-afb90678-4548-4a23-bbbb-3ce2ffb6b59c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-15-ecd3673c-c585-4fab-98ae-b14f5583e59d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-7-d534cf40-ecf6-4cac-a823-c5bb7d515ffe in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-15-6c51bd3b-88c0-407a-a606-f21f8a16b22e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-16-47e4622c-aea3-4a7f-b3c4-c4e88c553c63 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-13-6aef6aef-2fa7-4499-a138-8132f10662fd in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-1-d5563ddf-63dc-4308-b456-881b0f3bd64f in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-323295d3-d5b5-49f4-a364-18c681fcc534 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-494d66e9-5225-4673-8400-ebf1f96d5e5e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-1-5035e670-86ff-4365-9be1-daf28bb85894 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-7-2bccc8ef-31f0-4974-8042-4721d60d0be4 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-13-e5e10c2d-28a2-4def-a5ec-2e33e3162c6c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-4-b18fc59c-d2e0-474b-9eb7-76150001406a in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-1-c99e56c9-bd3b-4230-a3fa-c01ae0208333 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-13-c7331006-f001-4cee-b7f7-feb4be3db7e5 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-5-fcf52494-4825-43da-8b8f-622b69f19707 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-4-7f88af3b-bcc8-475c-84e0-e254871998b3 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-16-b5952574-6146-4f4f-9fb9-e5a1a0165296 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-0b751ea6-b228-489b-9679-381173ac7bb8 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-6-5f5ed961-0ac6-4463-8e2d-87a0d40de556 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-9-180c4643-92f2-4989-b807-b6a4f8ab4ac9 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-15-24048d2d-7442-4392-af17-b4a959e58bf7 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-5-6fc6f5b5-cd67-4cb6-a8fb-b4b4dd8fc2ea in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-8-9fca323f-e32d-4e4a-99eb-356557892607 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-3b403a81-75d5-44df-b9d5-c0accc173e67 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-15-dbd7b016-fb76-4ce0-be8e-753995fa701e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-12-39204442-c071-46d5-8f5d-45a89dbe9426 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-11-dd4283b7-9226-496c-834e-4abfba06396c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-2-f5f81416-0d70-4d44-86b4-cf834add0c5c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-12-a889fd7d-edc8-4323-9aea-4e7c98f24e64 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-6-cbe5c48d-24ac-4aab-87a8-227c1292bffa in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-2-5e1893ac-76ee-495c-9624-ee06b8004565 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-5-3f0832ba-0b9e-4ff1-8e10-1ba9c61a7b67 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-8-ab32a5e0-4781-487c-8e8f-2a163d1437dd in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-7-4457a612-a996-43ed-95ee-9a0e1c9e5371 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 02:47:09,911] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:41792-1811027 (kafka.network.Processor)
[2019-03-10 02:47:15,431] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10694 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:15,433] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10694 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:15,436] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10695 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:15,437] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10695 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:22,774] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10695 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:42,111] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10696 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:42,113] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10696 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:48:04,754] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:43668-1811310 (kafka.network.Processor)
[2019-03-10 02:48:10,975] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.153:56162-1811343 (kafka.network.Processor)
[2019-03-10 02:48:32,082] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:44598-1811455 (kafka.network.Processor)
[2019-03-10 02:50:09,387] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:48904-1811956 (kafka.network.Processor)
[2019-03-10 02:50:16,787] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10696 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:50:22,630] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10697 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:50:22,633] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10697 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:51:13,251] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10697 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:51:13,967] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10698 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:51:13,969] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10698 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:57:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 03:13:44,854] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.153:42480-1819319 (kafka.network.Processor)
[2019-03-10 03:16:11,256] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10698 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:16:39,196] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10699 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:16:39,204] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10699 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:16:39,221] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10699 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:17:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 03:17:31,693] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10700 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:17:31,695] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10700 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:20:05,312] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:60490-1821319 (kafka.network.Processor)
[2019-03-10 03:22:56,928] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:38542-1822186 (kafka.network.Processor)
[2019-03-10 03:24:43,989] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:42936-1822733 (kafka.network.Processor)
[2019-03-10 03:27:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 03:37:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
{code}
I've skipped logs about segment deletion and so on.
was (Author: kgn):
[~guozhang], I found server.log (there are 3 brokers, but only one of them has relevant log entries in log):
{code:text}
[2019-03-10 02:46:43,627] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10693 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:43,630] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10693 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:45,397] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10693 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-7-cc600ccb-72b6-4582-b288-3ea966f33c7c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-1-f284c5c8-22fc-40d5-aa30-46c2875463bd in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-5-e608adfc-e9ad-427e-99b5-99ff1538611d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-6-daa568cc-a5ee-49a9-aa7c-bd34fbb1da52 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-2-da70d65b-53e6-4467-b06d-63f02ff89eec in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-12-a0cb16ff-46eb-4945-a87a-244a20261af9 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-10-c10cb15d-edc3-4c6a-b21a-68e2b8298284 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-16-d80c8a16-be83-48f3-ac9d-08431b87c198 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-5-75899722-91a9-439a-a0c5-f23eb7c6c93f in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-11-14fe64e9-1553-49e1-abf9-5ea623847c84 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,628] INFO [GroupCoordinator 2]: Member consumer-3-9b62bde0-ff71-46f7-b6a4-7be14e6055b5 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-14-42336dad-be7c-49da-a76b-72d13a856943 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-6-1d57e91e-a6cd-4917-9560-19fa6579d02e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-13-a310873e-5c66-402c-a031-0b2fce8eb4cc in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-1-7c591ed8-01d8-4f2c-b756-a5c3d3f63eea in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-7-70c381bc-5d32-4e2c-923b-2c15c0f15aa2 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-15-5de2a785-0cd8-43f8-b3e2-6b1da2983910 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-10-92f6d7a8-cc6c-4091-9eb4-1a8be9a08541 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-5-bd971902-bea5-49ef-83f6-3788e7363924 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-11-18b7fff3-f477-44c8-bece-4bde079bb578 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-2-bb9ff074-6738-4092-a2c0-f3d49e5e7e2a in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-9-ef6b36b3-376f-4a04-a0cb-709c8a28712b in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-13-57c73f24-4e8d-407e-874b-ab95409f7b1c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-10-265e41ea-c8c3-47f2-96d2-33309f975f29 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-15-f3ea2d41-09df-454a-a13b-c2007e0bff32 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-11-dcbc5baf-b7e4-433f-9580-b5e846418748 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-14-3cfd686d-e590-4055-9c8c-70f673a0a624 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-8-43d4c7af-a8f5-4616-8346-33bed246a253 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-11-bf244902-001f-460c-8cf5-e9ac5e7cb4a6 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-10-74f336ca-6a0f-4603-b93a-fcee81953f9f in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-1-65fe1595-6bff-4506-8a9b-6f35f3663f48 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-16-470b7eb9-942d-48fa-b8c1-d128ae21900d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,629] INFO [GroupCoordinator 2]: Member consumer-2-d587e29f-4d7e-4234-a774-21813af82415 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-7-a1fbd1a5-a4a8-42a8-bf7e-d6742edb8003 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-16-47eaa6a5-1fe9-4216-8afa-aaa5a7684755 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-10-1cba9737-26c2-4879-82c7-d3ef6399a2b3 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-13-b4031cae-43c3-45a4-b258-b9cfe2645eb6 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-e23f1744-7781-4e09-b394-6e3a09a5ed6d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-4-59459730-4118-4379-a208-83bfba9a751e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-9-4291927f-119f-4610-bd00-f9503847ff05 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-6-b8d722e7-58d3-4e2f-af03-92ab01e23f8d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-14-b2504ede-535c-4884-aea5-03734661bb4d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-6-afb90678-4548-4a23-bbbb-3ce2ffb6b59c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-15-ecd3673c-c585-4fab-98ae-b14f5583e59d in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-7-d534cf40-ecf6-4cac-a823-c5bb7d515ffe in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-15-6c51bd3b-88c0-407a-a606-f21f8a16b22e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-16-47e4622c-aea3-4a7f-b3c4-c4e88c553c63 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-13-6aef6aef-2fa7-4499-a138-8132f10662fd in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-1-d5563ddf-63dc-4308-b456-881b0f3bd64f in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-323295d3-d5b5-49f4-a364-18c681fcc534 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-494d66e9-5225-4673-8400-ebf1f96d5e5e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-1-5035e670-86ff-4365-9be1-daf28bb85894 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-7-2bccc8ef-31f0-4974-8042-4721d60d0be4 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-13-e5e10c2d-28a2-4def-a5ec-2e33e3162c6c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-4-b18fc59c-d2e0-474b-9eb7-76150001406a in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-1-c99e56c9-bd3b-4230-a3fa-c01ae0208333 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-13-c7331006-f001-4cee-b7f7-feb4be3db7e5 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-5-fcf52494-4825-43da-8b8f-622b69f19707 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-4-7f88af3b-bcc8-475c-84e0-e254871998b3 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-16-b5952574-6146-4f4f-9fb9-e5a1a0165296 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-0b751ea6-b228-489b-9679-381173ac7bb8 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-6-5f5ed961-0ac6-4463-8e2d-87a0d40de556 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-9-180c4643-92f2-4989-b807-b6a4f8ab4ac9 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-15-24048d2d-7442-4392-af17-b4a959e58bf7 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-5-6fc6f5b5-cd67-4cb6-a8fb-b4b4dd8fc2ea in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-8-9fca323f-e32d-4e4a-99eb-356557892607 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-3-3b403a81-75d5-44df-b9d5-c0accc173e67 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-15-dbd7b016-fb76-4ce0-be8e-753995fa701e in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-12-39204442-c071-46d5-8f5d-45a89dbe9426 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-11-dd4283b7-9226-496c-834e-4abfba06396c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-2-f5f81416-0d70-4d44-86b4-cf834add0c5c in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,630] INFO [GroupCoordinator 2]: Member consumer-12-a889fd7d-edc8-4323-9aea-4e7c98f24e64 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-6-cbe5c48d-24ac-4aab-87a8-227c1292bffa in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-2-5e1893ac-76ee-495c-9624-ee06b8004565 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-5-3f0832ba-0b9e-4ff1-8e10-1ba9c61a7b67 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-8-ab32a5e0-4781-487c-8e8f-2a163d1437dd in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:46:53,631] INFO [GroupCoordinator 2]: Member consumer-7-4457a612-a996-43ed-95ee-9a0e1c9e5371 in group hercules.sink.elastic.legacy_logs_elk_c2 has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 02:47:09,911] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:41792-1811027 (kafka.network.Processor)
[2019-03-10 02:47:15,431] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10694 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:15,433] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10694 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:15,436] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10695 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:15,437] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10695 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:22,774] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10695 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:42,111] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10696 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:47:42,113] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10696 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:48:04,754] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:43668-1811310 (kafka.network.Processor)
[2019-03-10 02:48:10,975] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.153:56162-1811343 (kafka.network.Processor)
[2019-03-10 02:48:32,082] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:44598-1811455 (kafka.network.Processor)
[2019-03-10 02:50:09,387] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:48904-1811956 (kafka.network.Processor)
[2019-03-10 02:50:16,787] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10696 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:50:22,630] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10697 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:50:22,633] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10697 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:51:13,251] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10697 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:51:13,967] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10698 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:51:13,969] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10698 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 02:57:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 03:13:44,854] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.153:42480-1819319 (kafka.network.Processor)
[2019-03-10 03:16:11,256] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10698 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:16:39,196] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10699 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:16:39,204] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10699 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:16:39,221] INFO [GroupCoordinator 2]: Preparing to rebalance group hercules.sink.elastic.legacy_logs_elk_c2 with old generation 10699 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:17:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 03:17:31,693] INFO [GroupCoordinator 2]: Stabilized group hercules.sink.elastic.legacy_logs_elk_c2 generation 10700 (__consumer_offsets-25) (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:17:31,695] INFO [GroupCoordinator 2]: Assignment received from leader for group hercules.sink.elastic.legacy_logs_elk_c2 for generation 10700 (kafka.coordinator.group.GroupCoordinator)
[2019-03-10 03:20:05,312] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:60490-1821319 (kafka.network.Processor)
[2019-03-10 03:22:56,928] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:38542-1822186 (kafka.network.Processor)
[2019-03-10 03:24:43,989] WARN Attempting to send response via channel for which there is no open connection, connection id 10.0.3.151:9092-10.0.3.60:42936-1822733 (kafka.network.Processor)
[2019-03-10 03:27:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-10 03:37:00,306] INFO [GroupMetadataManager brokerId=2] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
{code}
I've skipped logs about segment deletion and so on.
> Consumer cannot rejoin to the group after rebalancing
> -----------------------------------------------------
>
> Key: KAFKA-8104
> URL: https://issues.apache.org/jira/browse/KAFKA-8104
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 2.0.0
> Reporter: Gregory Koshelev
> Priority: Critical
> Attachments: consumer-rejoin-fail.log
>
>
> TL;DR; {{KafkaConsumer}} cannot rejoin to the group due to inconsistent {{AbstractCoordinator.generation}} (which is {{NO_GENERATION}} and {{AbstractCoordinator.joinFuture}} (which is succeeded {{RequestFuture}}). See explanation below.
> There are 16 consumers in single process (threads from pool-4-thread-1 to pool-4-thread-16). All of them belong to single consumer group {{hercules.sink.elastic.legacy_logs_elk_c2}}. Rebalancing has been acquired and consumers have got {{CommitFailedException}} as expected:
> {noformat}
> 2019-03-10T03:16:37.023Z [pool-4-thread-10] WARN r.k.vostok.hercules.sink.SimpleSink - Commit failed due to rebalancing
> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:798)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:681)
> at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1334)
> at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1298)
> at ru.kontur.vostok.hercules.sink.Sink.commit(Sink.java:156)
> at ru.kontur.vostok.hercules.sink.SimpleSink.run(SimpleSink.java:104)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> After that, most of them successfully rejoined to the group with generation 10699:
> {noformat}
> 2019-03-10T03:16:39.208Z [pool-4-thread-13] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-13, groupId=hercules.sink.elastic.legacy_logs_elk_c2] Successfully joined group with generation 10699
> 2019-03-10T03:16:39.209Z [pool-4-thread-13] INFO o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-13, groupId=hercules.sink.elastic.legacy_logs_elk_c2] Setting newly assigned partitions [legacy_logs_elk_c2-18]
> ...
> 2019-03-10T03:16:39.216Z [pool-4-thread-11] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-11, groupId=hercules.sink.elastic.legacy_logs_elk_c2] Successfully joined group with generation 10699
> 2019-03-10T03:16:39.217Z [pool-4-thread-11] INFO o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-11, groupId=hercules.sink.elastic.legacy_logs_elk_c2] Setting newly assigned partitions [legacy_logs_elk_c2-10, legacy_logs_elk_c2-11]
> ...
> 2019-03-10T03:16:39.218Z [pool-4-thread-15] INFO o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=consumer-15, groupId=hercules.sink.elastic.legacy_logs_elk_c2] Setting newly assigned partitions [legacy_logs_elk_c2-24]
> 2019-03-10T03:16:42.320Z [kafka-coordinator-heartbeat-thread | hercules.sink.elastic.legacy_logs_elk_c2] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-6, groupId=hercules.sink.elastic.legacy_logs_elk_c2] Attempt to heartbeat failed since group is rebalancing
> 2019-03-10T03:16:42.320Z [kafka-coordinator-heartbeat-thread | hercules.sink.elastic.legacy_logs_elk_c2] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-5, groupId=hercules.sink.elastic.legacy_logs_elk_c2] Attempt to heartbeat failed since group is rebalancing
> 2019-03-10T03:16:42.323Z [kafka-coordinator-heartbeat-thread | hercules.sink.elastic.legacy_logs_elk_c2] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-7, groupId=hercules.sink.elastic.legacy_logs_elk_c2] Attempt to heartbeat failed since group is rebalancing
> 2019-03-10T03:17:13.235Z [pool-4-thread-4] INFO o.a.k.c.c.i.AbstractCoordinator - [Consumer clientId=consumer-4, groupId=hercules.sink.elastic.legacy_logs_elk_c2] Successfully joined group with generation -1
> {noformat}
> But one consumer (pool-4-thread-4) got strange generation -1 (see last log record from above).
> Further log records in attached log file.
> Finally, 15 consumers successfully rejoined. But consumer with thread {{pool-4-thread-4}} didn't rejoin:
> {noformat}
> 2019-03-10T03:17:13.355Z [pool-4-thread-4] ERROR r.k.vostok.hercules.sink.SimpleSink - Unspecified exception has been acquired
> java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:241)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:422)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:352)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:337)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:333)
> at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
> at ru.kontur.vostok.hercules.sink.Sink.poll(Sink.java:152)
> at ru.kontur.vostok.hercules.sink.SimpleSink.run(SimpleSink.java:70)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-03-10T03:17:13.360Z [pool-4-thread-4] ERROR r.k.vostok.hercules.sink.SimpleSink - Unspecified exception has been acquired
> java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:241)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:422)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:352)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:337)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:333)
> at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
> at ru.kontur.vostok.hercules.sink.Sink.poll(Sink.java:152)
> at ru.kontur.vostok.hercules.sink.SimpleSink.run(SimpleSink.java:70)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)}}
> {noformat}
> It is important to note, that {{KafkaConsumer.coordinator.joinFuture}} is not null and succeeded, but {{ConsumerCoordinator}} cannot perform {{resetJoinGroupFuture()}} due to exception was thrown from {{onJoinComplete()}}:
> {code:java}
> if (future.succeeded()) {
> // Duplicate the buffer in case `onJoinComplete` does not complete and needs to be retried.
> ByteBuffer memberAssignment = future.value().duplicate();
> onJoinComplete(generation.generationId, generation.memberId, generation.protocol, memberAssignment);
> // We reset the join group future only after the completion callback returns. This ensures
> // that if the callback is woken up, we will retry it on the next joinGroupIfNeeded.
> resetJoinGroupFuture();
> needsJoinPrepare = true;
> }
> {code}
> If I understood correctly, the generation was changed to {{NO_GENERATION}} in another thread by one of CoordinatorResponseHandlers.
>
> [^consumer-rejoin-fail.log]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)