You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shantanu Deshmukh <sh...@gmail.com> on 2018/08/09 07:00:09 UTC
Re: Very long consumer rebalances
I am facing too many problems these days. Now one of our consumer groups
is rebalancing every now and then. And rebalance takes very low, more than
5-10 minutes. Even after re-balancing I see that only half of the consumers
are active/receive assignment. Its all going haywire.
I am seeing these logs in kafka consumer logs. Can anyone help me
understand what is going on here? It is a very long piece of log, but
someone please help me. I am desperately looking for any solution since
more than 2 months now. But to no avail.
[2018-08-09 11:39:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:53] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:53] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:53] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:56] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:39:56] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:39:56] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:39:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
group bulk-email-consumer failed: The request timed out.
[2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
bulk-email-consumer
[2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
commit failed for group bulk-email-consumer: Offset commit failed with a
retriable exception. You should retry committing offsets.
[2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
coordinator request for group bulk-email-consumer to broker
10.189.179.117:9092 (id: 1 rack: null)
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
group bulk-email-consumer failed: The request timed out.
[2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
bulk-email-consumer
[2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
commit failed for group bulk-email-consumer: Offset commit failed with a
retriable exception. You should retry committing offsets.
[2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
coordinator request for group bulk-email-consumer to broker
10.189.179.118:9092 (id: 2 rack: null)
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
coordinator response ClientResponse(receivedTimeMs=1533795004158,
disconnected=false, request=ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
request=RequestSend(header={api_key=10,api_version=0,correlation_id=389784,client_id=consumer-26},
body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
sendTimeMs=1533795004157),
responseBody={error_code=0,coordinator={node_id=0,host=10.189.179.116,port=9092}})
[2018-08-09 11:40:04] :: INFO ::
AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
bulk-emai -consumer.
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
coordinator response ClientResponse(receivedTimeMs=1533795004718,
disconnected=false, request=ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
request=RequestSend(header={api_key=10,api_version=0,correlation_id=363859,client_id=consumer-25},
body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
sendTimeMs=1533795004145),
responseBody={error_code=0,coordinator={node_id=0,host=10.189.179.116,port=9092}})
[2018-08-09 11:40:04] :: INFO ::
AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
bulk-emai -consumer.
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:04] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:05] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:07] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:07] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:07] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:07] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:08] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:10] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:10] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:10] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:10] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:11] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:13] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:13] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:14] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:14] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:14] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:16] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:16] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:16] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:16] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:17] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:19] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:19] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:19] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:19] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:40:20] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:21] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
previously assigned partitions [bulk-email-6] for group bulk-email-consumer
[2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:22] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:22] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}} for group bulk-email-consumer
[2018-08-09 11:40:22] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:22] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:22] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
previously assigned partitions [bulk-email-9] for group bulk-email-consumer
[2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:22] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
previously assigned partitions [bulk-email-8] for group bulk-email-consumer
[2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:26] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 1
[2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster metadata
version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0 rack:
null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
[Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
[0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
[0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:28] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 1
[2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster metadata
version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2 rack:
null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
[Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
[0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
[0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
[2018-08-09 11:40:30] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:30] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:30] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:30] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:31] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:34] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 1
[2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster metadata
version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
[Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
[0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
[0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:45] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-0=OffsetAndMetadata{offset=25390438,
metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327, metadata=''},
bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for group
bulk-email-consumer
[2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto offset
commit failed for group bulk-email-consumer: Commit cannot be completed
since the group has already rebalanced and assigned the partitions to
another member. This means that the time between subsequent calls to poll()
was longer than the configured max.poll.interval.ms, which typically
implies that the poll loop is spending too much time message processing.
You can address this either by increasing the session timeout or by
reducing the maximum size of batches returned in poll() with
max.poll.records.
[2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
previously assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1]
for group bulk-email-consumer
[2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:50] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 0
[2018-08-09 11:40:50] :: DEBUG ::
ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
LEAVE_GROUP request ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
request=RequestSend(header={api_key=13,api_version=0,correlation_id=349313,client_id=consumer-19},
body={group_id=bulk-email-consumer,member_id=consumer-19-34903180-9cb3-496d-a913-b782476d9a52}),
createdTimeMs=1533794419195, sendTimeMs=1533794419195) with correlation id
349313 due to node 2147483647 being disconnected
[2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
bulk-email-consumer
[2018-08-09 11:40:50] :: DEBUG ::
ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
JOIN_GROUP request ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
request=RequestSend(header={api_key=11,api_version=1,correlation_id=349314,client_id=consumer-19},
body={group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
correlation id 349314 due to node 2147483647 being disconnected
[2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
bulk-email-consumer
[2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
coordinator request for group bulk-email-consumer to broker
10.189.179.118:9092 (id: 2 rack: null)
[2018-08-09 11:40:51] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
- Sending metadata request {topics=[bulk-email]} to node 2
[2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster metadata
version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
[Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
[0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
[0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
[0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
coordinator response ClientResponse(receivedTimeMs=1533795051063,
disconnected=false, request=ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
request=RequestSend(header={api_key=10,api_version=0,correlation_id=349316,client_id=consumer-19},
body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
sendTimeMs=1533795051061),
responseBody={error_code=0,coordinator={node_id=0,host=10.189.179.116,port=9092}})
[2018-08-09 11:40:51] :: INFO ::
AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
bulk-emai -consumer.
[2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 - (Re-)joining
group bulk-email-consumer
[2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
JoinGroup
({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=300000,member_id=,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647 rack:
null)
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:40:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:00] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:17] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:17] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:31] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:31] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:33] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:36] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:39] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:42] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:46] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:46] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:48] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:51] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:54] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:41:57] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:01] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:03] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:06] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:09] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:12] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:15] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:18] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:21] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:24] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:27] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:31] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:34] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:37] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:40] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:43] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:46] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:49] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:52] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:55] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:42:58] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:01] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:04] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:07] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:10] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:13] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:16] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:19] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:22] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:25] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
failed for group bulk-email-consumer since it is rebalancing.
[2018-08-09 11:43:26] :: DEBUG ::
AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful join
group response for group bulk-email-consumer:
{error_code=0,generation_id=1225,group_protocol=range,leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
[2018-08-09 11:43:26] :: DEBUG ::
AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful join
group response for group bulk-email-consumer:
{error_code=0,generation_id=1225,group_protocol=range,leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
[2018-08-09 11:43:26] :: DEBUG ::
AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful join
group response for group bulk-email-consumer:
{error_code=0,generation_id=1225,group_protocol=range,leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
[2018-08-09 11:43:26] :: DEBUG ::
AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful join
group response for group bulk-email-consumer:
{error_code=0,generation_id=1225,group_protocol=range,leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
lim=22
cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
lim=22
cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
lim=22
cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
lim=22 cap=22]}]}
[2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
follower SyncGroup for group bulk-email-consumer to coordinator
10.189.179.116:9092 (id: 2147483647 rack: null):
{group_id=bulk-email-consumer,generation_id=1225,member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
[2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
follower SyncGroup for group bulk-email-consumer to coordinator
10.189.179.116:9092 (id: 2147483647 rack: null):
{group_id=bulk-email-consumer,generation_id=1225,member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
[2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
follower SyncGroup for group bulk-email-consumer to coordinator
10.189.179.116:9092 (id: 2147483647 rack: null):
{group_id=bulk-email-consumer,generation_id=1225,member_id=consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 - Performing
assignment for group bulk-email-consumer using strategy range with
subscriptions
{consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=Subscription(topics=[bulk-email]),
consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=Subscription(topics=[bulk-email]),
consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=Subscription(topics=[bulk-email]),
consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=Subscription(topics=[bulk-email])}
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
assignment for group bulk-email-consumer:
{consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=Assignment(partitions=[bulk-email-3,
bulk-email-4, bulk-email-5]),
consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=Assignment(partitions=[bulk-email-0,
bulk-email-1, bulk-email-2]),
consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=Assignment(partitions=[bulk-email-6,
bulk-email-7]),
consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=Assignment(partitions=[bulk-email-8,
bulk-email-9])}
[2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending leader
SyncGroup for group bulk-email-consumer to coordinator 10.189.179.116:9092
(id: 2147483647 rack: null):
{group_id=bulk-email-consumer,generation_id=1225,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_assignment=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
lim=38
cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
lim=38
cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
lim=34
cap=34]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,member_assignment=java.nio.HeapByteBuffer[pos=0
lim=34 cap=34]}]}
[2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
joined group bulk-email-consumer with generation 1225
[2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
joined group bulk-email-consumer with generation 1225
[2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
joined group bulk-email-consumer with generation 1225
[2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
joined group bulk-email-consumer with generation 1225
[2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
assigned partitions [bulk-email-7, bulk-email-6] for group
bulk-email-consumer
[2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1] for group
bulk-email-consumer
[2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
assigned partitions [bulk-email-4, bulk-email-3, bulk-email-5] for group
bulk-email-consumer
[2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
assigned partitions [bulk-email-8, bulk-email-9] for group
bulk-email-consumer
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
bulk-email-consumer fetching committed offsets for partitions:
[bulk-email-7, bulk-email-6]
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
bulk-email-consumer fetching committed offsets for partitions:
[bulk-email-0, bulk-email-2, bulk-email-1]
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
bulk-email-consumer fetching committed offsets for partitions:
[bulk-email-4, bulk-email-3, bulk-email-5]
[2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
bulk-email-consumer fetching committed offsets for partitions:
[bulk-email-8, bulk-email-9]
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-7 to the committed offset 25435998
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-0 to the committed offset 25390842
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-6 to the committed offset 25463566
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-4 to the committed offset 25409183
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-8 to the committed offset 25465113
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-2 to the committed offset 25368327
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched records
for partition bulk-email-8 since it is no longer fetchable
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-3 to the committed offset 25422008
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-9 to the committed offset 25444488
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-1 to the committed offset 25369263
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
partition bulk-email-5 to the committed offset 25456518
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :470 - Ignoring fetched records
for bulk-email-0 at offset 25390442 since the current position is 25390842
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched records
for partition bulk-email-6 since it is no longer fetchable
[2018-08-09 11:43:26] :: DEBUG :: Fetcher :671 - Discarding stale fetch
response for partition bulk-email-1 since its offset 25369219 does not
match the expected offset 25369263
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25465113 for partition bulk-email-8
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25444488 for partition bulk-email-9
[2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
metadata=''}, bulk-email-9=OffsetAndMetadata{offset=25444488, metadata=''}}
for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25436018 for partition bulk-email-7
[2018-08-09 11:43:29] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25463566 for partition bulk-email-6
[2018-08-09 11:43:29] :: DEBUG ::
AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
heartbeat response for group bulk-email-consumer
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25409183 for partition bulk-email-4
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25422008 for partition bulk-email-3
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25456530 for partition bulk-email-5
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25390842 for partition bulk-email-0
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25368339 for partition bulk-email-2
[2018-08-09 11:43:29] :: DEBUG ::
ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
bulk-email-consumer committed offset 25369263 for partition bulk-email-1
[2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
autocommit of offsets {bulk-email-4=OffsetAndMetadata{offset=25409183,
metadata=''}, bulk-email-3=OffsetAndMetadata{offset=25422008, metadata=''},
bulk-email-5=OffsetAndMetadata{offset=25456530, metadata=''}} for group
bulk-email-consumer
On Thu, Jul 12, 2018 at 9:27 PM Steve Tian <st...@gmail.com> wrote:
> It's a very good and important doc so I think you should read it all. You
> should get some idea from sections like *Detecting Consumer Failures* and
> *Multi-threaded Processing* for your case.
>
> On Thu, Jul 12, 2018, 3:17 PM Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
> > Hi Steve,
> >
> > Could you please shed more light on this? What section should I revisit?
> I
> > am using high-level consumer. So I am simply calling consumer.close()
> when
> > I am shutting down the process. Is there any other method to be called
> > before calling close()?
> >
> > On Mon, Jul 9, 2018 at 5:58 PM Steve Tian <st...@gmail.com>
> wrote:
> >
> > > Please re-read the javadoc of KafkaConsumer, make sure you know how to
> > > wakeup/close consumer properly while shutting down your application.
> Try
> > > to understand the motivation of KIP-62 and adjust related timeout.
> > >
> > > On Mon, Jul 9, 2018, 8:05 PM harish lohar <hk...@gmail.com> wrote:
> > >
> > > > Try reducing below timer
> > > > metadata.max.age.ms = 300000
> > > >
> > > >
> > > > On Fri, Jul 6, 2018 at 5:55 AM Shantanu Deshmukh <
> > shantanu88d@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a java
> app
> > > > which
> > > > > spawns many consumer threads consuming from different topics. For
> > every
> > > > > topic we have specified different consumer-group. A lot of times I
> > see
> > > > that
> > > > > whenever this application is restarted a CG on one or two topics
> > takes
> > > > more
> > > > > than 5 minutes to receive partition assignment. Till that time
> > > consumers
> > > > > for that topic don't consumer anything. If I go to Kafka broker and
> > run
> > > > > consumer-groups.sh and describe that particular CG I see that it is
> > > > > rebalancing. There is time critical data stored in that topic and
> we
> > > > cannot
> > > > > tolerate such long delays. What can be the reason for such long
> > > > rebalances.
> > > > >
> > > > > Here's our consumer config
> > > > >
> > > > >
> > > > > auto.commit.interval.ms = 3000
> > > > > auto.offset.reset = latest
> > > > > bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092, x.x.x.x:9092]
> > > > > check.crcs = true
> > > > > client.id =
> > > > > connections.max.idle.ms = 540000
> > > > > enable.auto.commit = true
> > > > > exclude.internal.topics = true
> > > > > fetch.max.bytes = 52428800
> > > > > fetch.max.wait.ms = 500
> > > > > fetch.min.bytes = 1
> > > > > group.id = otp-notifications-consumer
> > > > > heartbeat.interval.ms = 3000
> > > > > interceptor.classes = null
> > > > > key.deserializer = class
> > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > max.partition.fetch.bytes = 1048576
> > > > > max.poll.interval.ms = 300000
> > > > > max.poll.records = 50
> > > > > metadata.max.age.ms = 300000
> > > > > metric.reporters = []
> > > > > metrics.num.samples = 2
> > > > > metrics.sample.window.ms = 30000
> > > > > partition.assignment.strategy = [class
> > > > > org.apache.kafka.clients.consumer.RangeAssignor]
> > > > > receive.buffer.bytes = 65536
> > > > > reconnect.backoff.ms = 50
> > > > > request.timeout.ms = 305000
> > > > > retry.backoff.ms = 100
> > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit
> > > > > sasl.kerberos.min.time.before.relogin = 60000
> > > > > sasl.kerberos.service.name = null
> > > > > sasl.kerberos.ticket.renew.jitter = 0.05
> > > > > sasl.kerberos.ticket.renew.window.factor = 0.8
> > > > > sasl.mechanism = GSSAPI
> > > > > security.protocol = SSL
> > > > > send.buffer.bytes = 131072
> > > > > session.timeout.ms = 300000
> > > > > ssl.cipher.suites = null
> > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> > > > > ssl.endpoint.identification.algorithm = null
> > > > > ssl.key.password = null
> > > > > ssl.keymanager.algorithm = SunX509
> > > > > ssl.keystore.location = null
> > > > > ssl.keystore.password = null
> > > > > ssl.keystore.type = JKS
> > > > > ssl.protocol = TLS
> > > > > ssl.provider = null
> > > > > ssl.secure.random.implementation = null
> > > > > ssl.trustmanager.algorithm = PKIX
> > > > > ssl.truststore.location = /x/x/client.truststore.jks
> > > > > ssl.truststore.password = [hidden]
> > > > > ssl.truststore.type = JKS
> > > > > value.deserializer = class
> > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > >
> > > > > Please help.
> > > > >
> > > > > *Thanks & Regards,*
> > > > > *Shantanu Deshmukh*
> > > > >
> > > >
> > >
> >
>
Re: Very long consumer rebalances
Posted by Shantanu Deshmukh <sh...@gmail.com>.
Can anyone help me understand how to debug this issue? I tried setting log
level to trace in consumer logback configuration. But at such times nothing
appears in log, even in trace level. It is like entire code is frozen.
On Thu, Aug 16, 2018 at 6:32 PM Shantanu Deshmukh <sh...@gmail.com>
wrote:
> I saw a few topics with segment.ms and retention.ms property set. Can
> that be causing any issue? I remember that this is the only change I
> carried out to the cluster in last couple of months after which the problem
> started.
>
> On Fri, Aug 10, 2018 at 2:55 PM M. Manna <ma...@gmail.com> wrote:
>
>> if you can upgrade, I would say upgrading to 0.10.2.x would be better for
>> you (or even higher, 2.0.0). Otherwise you have to play around with
>> max.poll.records and session.timeout.ms.
>>
>> As the doc says (or newer versions), the adjustment should be such that
>> request.timeout.ms >= max.poll.interval.ms. Also, heartbeat.interval.ms
>> should be curbed at (rule of thumb) 30% of session.timeout.ms.
>>
>> Lastly, all these have to be within the bounds of
>> group.min.session.timeout.ms and group.max.session.timeout.ms.
>>
>> You can check all these, tune them as necessary, and retry. Some of these
>> configs may or may not be applicable at runtime. so a rolling restart may
>> be required before all changes take place.
>>
>> On 9 August 2018 at 13:48, Shantanu Deshmukh <sh...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Yes my consumer application works like below
>> >
>> > 1. Reads how many workers are required to process each topics from
>> > properties file
>> > 2. As many threads are spawned as there are workers mentioned in
>> > properties file, topic name is passed to this thread. FixedThreadPool
>> > implementation is used.
>> > 3. Each worker thread initializes one consumer object and subscribes
>> to
>> > given topic. Consumer group is simply <topic-name>-consumer. So if my
>> > topic
>> > bulk-email, then consumer group for all those threads is
>> > bulk-email-consumer
>> > 4. Once this is done, inside an infinite while loop
>> consumer.poll(100)
>> > method keeps running. So this application is a daemon. Shuts down
>> only
>> > when
>> > server shuts down or in case of kill command.
>> >
>> > I have configured session.timeout.ms in consumer properties. I haven't
>> > done
>> > anything about zookeeper timeout. Is it required now? Since consumer
>> > accesses only the brokers.
>> >
>> > On Thu, Aug 9, 2018 at 3:03 PM M. Manna <ma...@gmail.com> wrote:
>> >
>> > > In the simplest way, how have you implemented your consumer?
>> > >
>> > > 1) Does your consumers join a designated group, process messages, and
>> > then
>> > > closes all connection? Or does it stay open perpetually until server
>> > > shutdown?
>> > > 2) Have you configured the session timeouts for client and zookeeper
>> > > accordingly?
>> > >
>> > > Regards,
>> > >
>> > > On 9 August 2018 at 08:00, Shantanu Deshmukh <sh...@gmail.com>
>> > > wrote:
>> > >
>> > > > I am facing too many problems these days. Now one of our consumer
>> > groups
>> > > > is rebalancing every now and then. And rebalance takes very low,
>> more
>> > > than
>> > > > 5-10 minutes. Even after re-balancing I see that only half of the
>> > > consumers
>> > > > are active/receive assignment. Its all going haywire.
>> > > >
>> > > > I am seeing these logs in kafka consumer logs. Can anyone help me
>> > > > understand what is going on here? It is a very long piece of log,
>> but
>> > > > someone please help me. I am desperately looking for any solution
>> since
>> > > > more than 2 months now. But to no avail.
>> > > >
>> > > > [2018-08-09 11:39:51] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:53] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:39:53] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:39:53] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:39:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:56] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:39:56] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:39:56] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:39:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:39:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:00] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:00] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:00] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:03] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:03] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:03] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
>> > for
>> > > > group bulk-email-consumer failed: The request timed out.
>> > > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking
>> the
>> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead
>> for
>> > > group
>> > > > bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
>> > offset
>> > > > commit failed for group bulk-email-consumer: Offset commit failed
>> with
>> > a
>> > > > retriable exception. You should retry committing offsets.
>> > > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
>> > > > coordinator request for group bulk-email-consumer to broker
>> > > > 10.189.179.117:9092 (id: 1 rack: null)
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
>> > for
>> > > > group bulk-email-consumer failed: The request timed out.
>> > > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking
>> the
>> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead
>> for
>> > > group
>> > > > bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
>> > offset
>> > > > commit failed for group bulk-email-consumer: Offset commit failed
>> with
>> > a
>> > > > retriable exception. You should retry committing offsets.
>> > > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
>> > > > coordinator request for group bulk-email-consumer to broker
>> > > > 10.189.179.118:9092 (id: 2 rack: null)
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
>> > group
>> > > > coordinator response ClientResponse(receivedTimeMs=1533795004158,
>> > > > disconnected=false, request=ClientRequest(expectResponse=true,
>> > > > callback=org.apache.kafka.clients.consumer.internals.
>> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
>> > > > request=RequestSend(header={api_key=10,api_version=0,
>> > > > correlation_id=389784,client_id=consumer-26},
>> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
>> > > > sendTimeMs=1533795004157),
>> > > > responseBody={error_code=0,coordinator={node_id=0,host=
>> > > > 10.189.179.116,port=9092}})
>> > > > [2018-08-09 11:40:04] :: INFO ::
>> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
>> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
>> group
>> > > > bulk-emai -consumer.
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
>> > group
>> > > > coordinator response ClientResponse(receivedTimeMs=1533795004718,
>> > > > disconnected=false, request=ClientRequest(expectResponse=true,
>> > > > callback=org.apache.kafka.clients.consumer.internals.
>> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
>> > > > request=RequestSend(header={api_key=10,api_version=0,
>> > > > correlation_id=363859,client_id=consumer-25},
>> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
>> > > > sendTimeMs=1533795004145),
>> > > > responseBody={error_code=0,coordinator={node_id=0,host=
>> > > > 10.189.179.116,port=9092}})
>> > > > [2018-08-09 11:40:04] :: INFO ::
>> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
>> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
>> group
>> > > > bulk-emai -consumer.
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:40:04] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:05] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:06] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:06] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:06] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:06] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:06] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:06] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:07] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:07] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:07] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:07] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:08] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:09] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:09] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:09] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:09] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:09] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:09] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:10] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:10] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:10] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:10] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:11] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:12] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:12] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:12] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:12] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:12] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:12] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:13] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:13] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:14] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:40:14] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:14] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:15] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:15] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:15] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:15] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:15] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:15] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:16] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:16] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:16] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:16] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:17] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:18] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:18] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:18] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:18] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:18] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:18] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:19] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:19] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:19] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:19] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
>> successful
>> > > > heartbeat response for group bulk-email-consumer
>> > > > [2018-08-09 11:40:20] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-6=OffsetAndMetadata{offset=25463566,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:21] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:21] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:21] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:21] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:21] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:21] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:21] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25463566 for partition
>> > bulk-email-6
>> > > > [2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
>> > > > previously assigned partitions [bulk-email-6] for group
>> > > bulk-email-consumer
>> > > > [2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 -
>> > (Re-)joining
>> > > > group bulk-email-consumer
>> > > > [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
>> > > > JoinGroup
>> > > > ({group_id=bulk-email-consumer,session_timeout=
>> > 300000,rebalance_timeout=
>> > > > 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
>> > > > 869377f9ead1,protocol_type=consumer,group_protocols=[{
>> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
>> 2147483647
>> > > > rack:
>> > > > null)
>> > > > [2018-08-09 11:40:22] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-9=OffsetAndMetadata{offset=25444488,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:22] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-8=OffsetAndMetadata{offset=25465113,
>> > > > metadata=''}} for group bulk-email-consumer
>> > > > [2018-08-09 11:40:22] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:22] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:22] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25444488 for partition
>> > bulk-email-9
>> > > > [2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
>> > > > previously assigned partitions [bulk-email-9] for group
>> > > bulk-email-consumer
>> > > > [2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 -
>> > (Re-)joining
>> > > > group bulk-email-consumer
>> > > > [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
>> > > > JoinGroup
>> > > > ({group_id=bulk-email-consumer,session_timeout=
>> > 300000,rebalance_timeout=
>> > > > 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
>> > > > 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
>> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
>> 2147483647
>> > > > rack:
>> > > > null)
>> > > > [2018-08-09 11:40:22] :: DEBUG ::
>> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
>> > > > bulk-email-consumer committed offset 25465113 for partition
>> > bulk-email-8
>> > > > [2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
>> > > > previously assigned partitions [bulk-email-8] for group
>> > > bulk-email-consumer
>> > > > [2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 -
>> > (Re-)joining
>> > > > group bulk-email-consumer
>> > > > [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
>> > > > JoinGroup
>> > > > ({group_id=bulk-email-consumer,session_timeout=
>> > 300000,rebalance_timeout=
>> > > > 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
>> > > > 9d083b256ffa,protocol_type=consumer,group_protocols=[{
>> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
>> 2147483647
>> > > > rack:
>> > > > null)
>> > > > [2018-08-09 11:40:24] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:24] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:24] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:24] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:24] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:26] :: DEBUG ::
>> > > NetworkClient$DefaultMetadataUpdater:640
>> > > > - Sending metadata request {topics=[bulk-email]} to node 1
>> > > > [2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster
>> > > metadata
>> > > > version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
>> > > > 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0
>> > rack:
>> > > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
>> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
>> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition
>> = 7,
>> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
>> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
>> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
>> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
>> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
>> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
>> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition
>> = 5,
>> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
>> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
>> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
>> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
>> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
>> > > > [2018-08-09 11:40:27] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:27] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:27] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:27] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:27] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:28] :: DEBUG ::
>> > > NetworkClient$DefaultMetadataUpdater:640
>> > > > - Sending metadata request {topics=[bulk-email]} to node 1
>> > > > [2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster
>> > > metadata
>> > > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
>> > > > 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2
>> > rack:
>> > > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
>> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
>> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition
>> = 7,
>> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
>> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
>> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
>> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
>> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
>> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
>> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition
>> = 5,
>> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
>> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
>> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
>> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
>> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
>> > > > [2018-08-09 11:40:30] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:30] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:30] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:30] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:31] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:33] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:33] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:33] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:33] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:33] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:34] :: DEBUG ::
>> > > NetworkClient$DefaultMetadataUpdater:640
>> > > > - Sending metadata request {topics=[bulk-email]} to node 1
>> > > > [2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster
>> > > metadata
>> > > > version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
>> > > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
>> > rack:
>> > > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
>> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
>> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition
>> = 7,
>> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
>> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
>> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
>> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
>> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
>> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
>> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition
>> = 5,
>> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
>> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
>> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
>> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
>> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
>> > > > [2018-08-09 11:40:36] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:36] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:36] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:36] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:36] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:39] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:39] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:39] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:39] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:39] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:42] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:42] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:42] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:42] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:42] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:45] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:45] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:45] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:45] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:45] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:48] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:48] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:48] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:48] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:48] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 -
>> Completed
>> > > > autocommit of offsets
>> {bulk-email-0=OffsetAndMetadata{offset=25390438,
>> > > > metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
>> > > > metadata=''},
>> > > > bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for
>> > group
>> > > > bulk-email-consumer
>> > > > [2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto
>> offset
>> > > > commit failed for group bulk-email-consumer: Commit cannot be
>> completed
>> > > > since the group has already rebalanced and assigned the partitions
>> to
>> > > > another member. This means that the time between subsequent calls to
>> > > poll()
>> > > > was longer than the configured max.poll.interval.ms, which
>> typically
>> > > > implies that the poll loop is spending too much time message
>> > processing.
>> > > > You can address this either by increasing the session timeout or by
>> > > > reducing the maximum size of batches returned in poll() with
>> > > > max.poll.records.
>> > > > [2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
>> > > > previously assigned partitions [bulk-email-0, bulk-email-2,
>> > bulk-email-1]
>> > > > for group bulk-email-consumer
>> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 -
>> > (Re-)joining
>> > > > group bulk-email-consumer
>> > > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
>> > > > JoinGroup
>> > > > ({group_id=bulk-email-consumer,session_timeout=
>> > 300000,rebalance_timeout=
>> > > > 300000,member_id=,protocol_type=consumer,group_protocols=
>> > > >
>> [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
>> 2147483647
>> > > > rack:
>> > > > null)
>> > > > [2018-08-09 11:40:50] :: DEBUG ::
>> > > NetworkClient$DefaultMetadataUpdater:640
>> > > > - Sending metadata request {topics=[bulk-email]} to node 0
>> > > > [2018-08-09 11:40:50] :: DEBUG ::
>> > > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
>> > > > LEAVE_GROUP request ClientRequest(expectResponse=true,
>> > > > callback=org.apache.kafka.clients.consumer.internals.
>> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
>> > > > request=RequestSend(header={api_key=13,api_version=0,
>> > > > correlation_id=349313,client_id=consumer-19},
>> > > > body={group_id=bulk-email-consumer,member_id=consumer-
>> > > > 19-34903180-9cb3-496d-a913-b782476d9a52}),
>> > > > createdTimeMs=1533794419195, sendTimeMs=1533794419195) with
>> correlation
>> > > id
>> > > > 349313 due to node 2147483647 being disconnected
>> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking
>> the
>> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead
>> for
>> > > group
>> > > > bulk-email-consumer
>> > > > [2018-08-09 11:40:50] :: DEBUG ::
>> > > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
>> > > > JOIN_GROUP request ClientRequest(expectResponse=true,
>> > > > callback=org.apache.kafka.clients.consumer.internals.
>> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
>> > > > request=RequestSend(header={api_key=11,api_version=1,
>> > > > correlation_id=349314,client_id=consumer-19},
>> > > > body={group_id=bulk-email-consumer,session_timeout=
>> > > > 300000,rebalance_timeout=300000,member_id=,protocol_
>> > > > type=consumer,group_protocols=[{protocol_name=range,
>> > > > protocol_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
>> > > > correlation id 349314 due to node 2147483647 being disconnected
>> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking
>> the
>> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead
>> for
>> > > group
>> > > > bulk-email-consumer
>> > > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
>> > > > coordinator request for group bulk-email-consumer to broker
>> > > > 10.189.179.118:9092 (id: 2 rack: null)
>> > > > [2018-08-09 11:40:51] :: DEBUG ::
>> > > NetworkClient$DefaultMetadataUpdater:640
>> > > > - Sending metadata request {topics=[bulk-email]} to node 2
>> > > > [2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster
>> > > metadata
>> > > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
>> > > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
>> > rack:
>> > > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
>> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
>> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition
>> = 7,
>> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
>> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
>> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
>> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
>> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
>> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
>> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition
>> = 5,
>> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
>> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
>> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
>> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
>> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
>> > > > [2018-08-09 11:40:51] :: DEBUG ::
>> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
>> > group
>> > > > coordinator response ClientResponse(receivedTimeMs=1533795051063,
>> > > > disconnected=false, request=ClientRequest(expectResponse=true,
>> > > > callback=org.apache.kafka.clients.consumer.internals.
>> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
>> > > > request=RequestSend(header={api_key=10,api_version=0,
>> > > > correlation_id=349316,client_id=consumer-19},
>> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
>> > > > sendTimeMs=1533795051061),
>> > > > responseBody={error_code=0,coordinator={node_id=0,host=
>> > > > 10.189.179.116,port=9092}})
>> > > > [2018-08-09 11:40:51] :: INFO ::
>> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
>> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
>> group
>> > > > bulk-emai -consumer.
>> > > > [2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 -
>> > (Re-)joining
>> > > > group bulk-email-consumer
>> > > > [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
>> > > > JoinGroup
>> > > > ({group_id=bulk-email-consumer,session_timeout=
>> > 300000,rebalance_timeout=
>> > > > 300000,member_id=,protocol_type=consumer,group_protocols=
>> > > >
>> [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
>> 2147483647
>> > > > rack:
>> > > > null)
>> > > > [2018-08-09 11:40:51] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:51] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:51] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:51] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:40:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:00] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:00] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:03] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:03] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:06] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:06] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:09] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:09] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:12] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:15] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:17] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:17] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:18] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:18] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:21] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:21] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:24] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:24] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:27] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:27] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:31] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:31] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:33] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:33] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:36] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:36] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:39] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:39] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:42] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:42] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:46] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:46] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:48] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:48] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:51] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:51] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:54] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:41:57] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:01] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:03] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:06] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:09] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:12] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:15] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:18] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:21] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:24] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:27] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:31] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:34] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:37] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:40] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:43] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:46] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:49] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:52] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:55] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:42:58] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:01] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:04] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:07] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:10] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:13] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:16] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:19] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:22] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:25] :: DEBUG ::
>> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
>> > beat
>> > > > failed for group bulk-email-consumer since it is rebalancing.
>> > > > [2018-08-09 11:43:26] :: DEBUG ::
>> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
>> successful
>> > > > join
>> > > > group response for group bulk-email-consumer:
>> > > > {error_code=0,generation_id=1225,group_protocol=range,
>> > > >
>> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
>> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
>> > > > [2018-08-09 11:43:26] :: DEBUG ::
>> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
>> successful
>> > > > join
>> > > > group response for group bulk-email-consumer:
>> > > > {error_code=0,generation_id=1225,group_protocol=range,
>> > > >
>> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
>> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
>> > > > [2018-08-09 11:43:26] :: DEBUG ::
>> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
>> successful
>> > > > join
>> > > > group response for group bulk-email-consumer:
>> > > > {error_code=0,generation_id=1225,group_protocol=range,
>> > > >
>> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
>> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
>> > > > [2018-08-09 11:43:26] :: DEBUG ::
>> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
>> successful
>> > > > join
>> > > > group response for group bulk-email-consumer:
>> > > > {error_code=0,generation_id=1225,group_protocol=range,
>> > > >
>> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
>> > > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
>> > > > members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
>> > > > 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22
>> > > > cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
>> > > > 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22
>> > > > cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
>> > > > 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22
>> > > > cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
>> > > > 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
>> > > > lim=22 cap=22]}]}
>> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
>> > > > follower SyncGroup for group bulk-email-consumer to coordinator
>> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
>> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
>> > > >
>> consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
>> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
>> > > > follower SyncGroup for group bulk-email-consumer to coordinator
>> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
>> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
>> > > >
>> consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
>> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
>> > > > follower SyncGroup for group bulk-email-consumer to coordinator
>> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
>> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
>> > > >
>> consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
>> > > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 -
>> Performing
>> > > > assignment for group bulk-email-consumer using strategy range with
>> > > > subscriptions
>> > > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
>> > > > Subscription(topics=[bulk-email]),
>> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
>> > > > Subscription(topics=[bulk-email]),
>> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
>> > > > Subscription(topics=[bulk-email]),
>> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
>> > > > Subscription(topics=[bulk-email])}
>> > > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
>> > > > assignment for group bulk-email-consumer:
>> > > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
>> > > > Assignment(partitions=[bulk-email-3,
>> > > > bulk-email-4, bulk-email-5]),
>> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
>> > > > Assignment(partitions=[bulk-email-0,
>> > > > bulk-email-1, bulk-email-2]),
>> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
>> > > > Assignment(partitions=[bulk-email-6,
>> > > > bulk-email-7]),
>> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
>> > > > Assignment(partitions=[bulk-email-8,
>> > > > bulk-email-9])}
>> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending
>> > > leader
>> > > > SyncGroup for group bulk-email-consumer to coordinator
>> > > 10.189.179.116:9092
>> > > > (id: 2147483647 rack: null):
>> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
>> > > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
>> > > > assignment=[{member_id=consumer-23-bdc073b6-1f90-
>> > > >
>> 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
>> > > > lim=38
>> > > > cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
>> > > > 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
>> > > > lim=38
>> > > > cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
>> > > > 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
>> > > > lim=34
>> > > > cap=34]},{member_id=consumer-26
>
>
Re: Very long consumer rebalances
Posted by Shantanu Deshmukh <sh...@gmail.com>.
I saw a few topics with segment.ms and retention.ms property set. Can that
be causing any issue? I remember that this is the only change I carried out
to the cluster in last couple of months after which the problem started.
On Fri, Aug 10, 2018 at 2:55 PM M. Manna <ma...@gmail.com> wrote:
> if you can upgrade, I would say upgrading to 0.10.2.x would be better for
> you (or even higher, 2.0.0). Otherwise you have to play around with
> max.poll.records and session.timeout.ms.
>
> As the doc says (or newer versions), the adjustment should be such that
> request.timeout.ms >= max.poll.interval.ms. Also, heartbeat.interval.ms
> should be curbed at (rule of thumb) 30% of session.timeout.ms.
>
> Lastly, all these have to be within the bounds of
> group.min.session.timeout.ms and group.max.session.timeout.ms.
>
> You can check all these, tune them as necessary, and retry. Some of these
> configs may or may not be applicable at runtime. so a rolling restart may
> be required before all changes take place.
>
> On 9 August 2018 at 13:48, Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Yes my consumer application works like below
> >
> > 1. Reads how many workers are required to process each topics from
> > properties file
> > 2. As many threads are spawned as there are workers mentioned in
> > properties file, topic name is passed to this thread. FixedThreadPool
> > implementation is used.
> > 3. Each worker thread initializes one consumer object and subscribes
> to
> > given topic. Consumer group is simply <topic-name>-consumer. So if my
> > topic
> > bulk-email, then consumer group for all those threads is
> > bulk-email-consumer
> > 4. Once this is done, inside an infinite while loop consumer.poll(100)
> > method keeps running. So this application is a daemon. Shuts down only
> > when
> > server shuts down or in case of kill command.
> >
> > I have configured session.timeout.ms in consumer properties. I haven't
> > done
> > anything about zookeeper timeout. Is it required now? Since consumer
> > accesses only the brokers.
> >
> > On Thu, Aug 9, 2018 at 3:03 PM M. Manna <ma...@gmail.com> wrote:
> >
> > > In the simplest way, how have you implemented your consumer?
> > >
> > > 1) Does your consumers join a designated group, process messages, and
> > then
> > > closes all connection? Or does it stay open perpetually until server
> > > shutdown?
> > > 2) Have you configured the session timeouts for client and zookeeper
> > > accordingly?
> > >
> > > Regards,
> > >
> > > On 9 August 2018 at 08:00, Shantanu Deshmukh <sh...@gmail.com>
> > > wrote:
> > >
> > > > I am facing too many problems these days. Now one of our consumer
> > groups
> > > > is rebalancing every now and then. And rebalance takes very low, more
> > > than
> > > > 5-10 minutes. Even after re-balancing I see that only half of the
> > > consumers
> > > > are active/receive assignment. Its all going haywire.
> > > >
> > > > I am seeing these logs in kafka consumer logs. Can anyone help me
> > > > understand what is going on here? It is a very long piece of log, but
> > > > someone please help me. I am desperately looking for any solution
> since
> > > > more than 2 months now. But to no avail.
> > > >
> > > > [2018-08-09 11:39:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:53] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:53] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:53] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:56] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:39:56] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:39:56] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> > for
> > > > group bulk-email-consumer failed: The request timed out.
> > > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> > offset
> > > > commit failed for group bulk-email-consumer: Offset commit failed
> with
> > a
> > > > retriable exception. You should retry committing offsets.
> > > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > > coordinator request for group bulk-email-consumer to broker
> > > > 10.189.179.117:9092 (id: 1 rack: null)
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> > for
> > > > group bulk-email-consumer failed: The request timed out.
> > > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> > offset
> > > > commit failed for group bulk-email-consumer: Offset commit failed
> with
> > a
> > > > retriable exception. You should retry committing offsets.
> > > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > > coordinator request for group bulk-email-consumer to broker
> > > > 10.189.179.118:9092 (id: 2 rack: null)
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> > group
> > > > coordinator response ClientResponse(receivedTimeMs=1533795004158,
> > > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
> > > > request=RequestSend(header={api_key=10,api_version=0,
> > > > correlation_id=389784,client_id=consumer-26},
> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
> > > > sendTimeMs=1533795004157),
> > > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > > 10.189.179.116,port=9092}})
> > > > [2018-08-09 11:40:04] :: INFO ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
> group
> > > > bulk-emai -consumer.
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> > group
> > > > coordinator response ClientResponse(receivedTimeMs=1533795004718,
> > > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
> > > > request=RequestSend(header={api_key=10,api_version=0,
> > > > correlation_id=363859,client_id=consumer-25},
> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
> > > > sendTimeMs=1533795004145),
> > > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > > 10.189.179.116,port=9092}})
> > > > [2018-08-09 11:40:04] :: INFO ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
> group
> > > > bulk-emai -consumer.
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:05] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:08] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:11] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:13] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:13] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:14] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:14] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:14] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:17] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:20] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-6] for group
> > > bulk-email-consumer
> > > > [2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > > 869377f9ead1,protocol_type=consumer,group_protocols=[{
> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-9] for group
> > > bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > > 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-8] for group
> > > bulk-email-consumer
> > > > [2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > > 9d083b256ffa,protocol_type=consumer,group_protocols=[{
> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:26] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > > [2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0
> > rack:
> > > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:28] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > > [2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2
> > rack:
> > > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:34] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > > [2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> > rack:
> > > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-0=OffsetAndMetadata{offset=25390438,
> > > > metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
> > > > metadata=''},
> > > > bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for
> > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto
> offset
> > > > commit failed for group bulk-email-consumer: Commit cannot be
> completed
> > > > since the group has already rebalanced and assigned the partitions to
> > > > another member. This means that the time between subsequent calls to
> > > poll()
> > > > was longer than the configured max.poll.interval.ms, which typically
> > > > implies that the poll loop is spending too much time message
> > processing.
> > > > You can address this either by increasing the session timeout or by
> > > > reducing the maximum size of batches returned in poll() with
> > > > max.poll.records.
> > > > [2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-0, bulk-email-2,
> > bulk-email-1]
> > > > for group bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:50] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 0
> > > > [2018-08-09 11:40:50] :: DEBUG ::
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > > LEAVE_GROUP request ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
> > > > request=RequestSend(header={api_key=13,api_version=0,
> > > > correlation_id=349313,client_id=consumer-19},
> > > > body={group_id=bulk-email-consumer,member_id=consumer-
> > > > 19-34903180-9cb3-496d-a913-b782476d9a52}),
> > > > createdTimeMs=1533794419195, sendTimeMs=1533794419195) with
> correlation
> > > id
> > > > 349313 due to node 2147483647 being disconnected
> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: DEBUG ::
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > > JOIN_GROUP request ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
> > > > request=RequestSend(header={api_key=11,api_version=1,
> > > > correlation_id=349314,client_id=consumer-19},
> > > > body={group_id=bulk-email-consumer,session_timeout=
> > > > 300000,rebalance_timeout=300000,member_id=,protocol_
> > > > type=consumer,group_protocols=[{protocol_name=range,
> > > > protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
> > > > correlation id 349314 due to node 2147483647 being disconnected
> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > > coordinator request for group bulk-email-consumer to broker
> > > > 10.189.179.118:9092 (id: 2 rack: null)
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 2
> > > > [2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> > rack:
> > > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> > group
> > > > coordinator response ClientResponse(receivedTimeMs=1533795051063,
> > > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
> > > > request=RequestSend(header={api_key=10,api_version=0,
> > > > correlation_id=349316,client_id=consumer-19},
> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
> > > > sendTimeMs=1533795051061),
> > > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > > 10.189.179.116,port=9092}})
> > > > [2018-08-09 11:40:51] :: INFO ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
> group
> > > > bulk-emai -consumer.
> > > > [2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:17] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:17] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:46] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:46] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:01] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:34] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:37] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:40] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:43] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:46] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:49] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:52] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:55] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:58] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:01] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:07] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:10] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:13] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:16] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:19] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:22] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:25] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
> > > > members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > > 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22
> > > > cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > > 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22
> > > > cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > > 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22
> > > > cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > > 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 -
> Performing
> > > > assignment for group bulk-email-consumer using strategy range with
> > > > subscriptions
> > > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > > Subscription(topics=[bulk-email]),
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > > Subscription(topics=[bulk-email]),
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > > Subscription(topics=[bulk-email]),
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > > Subscription(topics=[bulk-email])}
> > > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
> > > > assignment for group bulk-email-consumer:
> > > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > > Assignment(partitions=[bulk-email-3,
> > > > bulk-email-4, bulk-email-5]),
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > > Assignment(partitions=[bulk-email-0,
> > > > bulk-email-1, bulk-email-2]),
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > > Assignment(partitions=[bulk-email-6,
> > > > bulk-email-7]),
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > > Assignment(partitions=[bulk-email-8,
> > > > bulk-email-9])}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending
> > > leader
> > > > SyncGroup for group bulk-email-consumer to coordinator
> > > 10.189.179.116:9092
> > > > (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
> > > > assignment=[{member_id=consumer-23-bdc073b6-1f90-
> > > >
> 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > > lim=38
> > > > cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > > 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > > lim=38
> > > > cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > > 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > > lim=34
> > > > cap=34]},{member_id=consumer-26
Re: Very long consumer rebalances
Posted by Shantanu Deshmukh <sh...@gmail.com>.
All this is fine. But what I do not understand is, why only some of the
consumer groups start very late. We have 7 topics and their consumers
belong to 7 different CGs. Whenever I want to restart my application Only
one or two of them will start very late. It takes almost 5-10 minutes
before their consumers start receiving data.
When that delay occurs there is absolutely no logs generated by consumer
library.
On Fri, Aug 10, 2018 at 2:55 PM M. Manna <ma...@gmail.com> wrote:
> if you can upgrade, I would say upgrading to 0.10.2.x would be better for
> you (or even higher, 2.0.0). Otherwise you have to play around with
> max.poll.records and session.timeout.ms.
>
> As the doc says (or newer versions), the adjustment should be such that
> request.timeout.ms >= max.poll.interval.ms. Also, heartbeat.interval.ms
> should be curbed at (rule of thumb) 30% of session.timeout.ms.
>
> Lastly, all these have to be within the bounds of
> group.min.session.timeout.ms and group.max.session.timeout.ms.
>
> You can check all these, tune them as necessary, and retry. Some of these
> configs may or may not be applicable at runtime. so a rolling restart may
> be required before all changes take place.
>
> On 9 August 2018 at 13:48, Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Yes my consumer application works like below
> >
> > 1. Reads how many workers are required to process each topics from
> > properties file
> > 2. As many threads are spawned as there are workers mentioned in
> > properties file, topic name is passed to this thread. FixedThreadPool
> > implementation is used.
> > 3. Each worker thread initializes one consumer object and subscribes
> to
> > given topic. Consumer group is simply <topic-name>-consumer. So if my
> > topic
> > bulk-email, then consumer group for all those threads is
> > bulk-email-consumer
> > 4. Once this is done, inside an infinite while loop consumer.poll(100)
> > method keeps running. So this application is a daemon. Shuts down only
> > when
> > server shuts down or in case of kill command.
> >
> > I have configured session.timeout.ms in consumer properties. I haven't
> > done
> > anything about zookeeper timeout. Is it required now? Since consumer
> > accesses only the brokers.
> >
> > On Thu, Aug 9, 2018 at 3:03 PM M. Manna <ma...@gmail.com> wrote:
> >
> > > In the simplest way, how have you implemented your consumer?
> > >
> > > 1) Does your consumers join a designated group, process messages, and
> > then
> > > closes all connection? Or does it stay open perpetually until server
> > > shutdown?
> > > 2) Have you configured the session timeouts for client and zookeeper
> > > accordingly?
> > >
> > > Regards,
> > >
> > > On 9 August 2018 at 08:00, Shantanu Deshmukh <sh...@gmail.com>
> > > wrote:
> > >
> > > > I am facing too many problems these days. Now one of our consumer
> > groups
> > > > is rebalancing every now and then. And rebalance takes very low, more
> > > than
> > > > 5-10 minutes. Even after re-balancing I see that only half of the
> > > consumers
> > > > are active/receive assignment. Its all going haywire.
> > > >
> > > > I am seeing these logs in kafka consumer logs. Can anyone help me
> > > > understand what is going on here? It is a very long piece of log, but
> > > > someone please help me. I am desperately looking for any solution
> since
> > > > more than 2 months now. But to no avail.
> > > >
> > > > [2018-08-09 11:39:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:53] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:53] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:53] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:56] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:39:56] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:39:56] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:39:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> > for
> > > > group bulk-email-consumer failed: The request timed out.
> > > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> > offset
> > > > commit failed for group bulk-email-consumer: Offset commit failed
> with
> > a
> > > > retriable exception. You should retry committing offsets.
> > > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > > coordinator request for group bulk-email-consumer to broker
> > > > 10.189.179.117:9092 (id: 1 rack: null)
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> > for
> > > > group bulk-email-consumer failed: The request timed out.
> > > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> > offset
> > > > commit failed for group bulk-email-consumer: Offset commit failed
> with
> > a
> > > > retriable exception. You should retry committing offsets.
> > > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > > coordinator request for group bulk-email-consumer to broker
> > > > 10.189.179.118:9092 (id: 2 rack: null)
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> > group
> > > > coordinator response ClientResponse(receivedTimeMs=1533795004158,
> > > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
> > > > request=RequestSend(header={api_key=10,api_version=0,
> > > > correlation_id=389784,client_id=consumer-26},
> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
> > > > sendTimeMs=1533795004157),
> > > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > > 10.189.179.116,port=9092}})
> > > > [2018-08-09 11:40:04] :: INFO ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
> group
> > > > bulk-emai -consumer.
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> > group
> > > > coordinator response ClientResponse(receivedTimeMs=1533795004718,
> > > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
> > > > request=RequestSend(header={api_key=10,api_version=0,
> > > > correlation_id=363859,client_id=consumer-25},
> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
> > > > sendTimeMs=1533795004145),
> > > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > > 10.189.179.116,port=9092}})
> > > > [2018-08-09 11:40:04] :: INFO ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
> group
> > > > bulk-emai -consumer.
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:04] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:05] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:07] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:08] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:10] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:11] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:13] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:13] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:14] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:14] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:14] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:16] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:17] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:19] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received
> successful
> > > > heartbeat response for group bulk-email-consumer
> > > > [2018-08-09 11:40:20] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:21] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25463566 for partition
> > bulk-email-6
> > > > [2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-6] for group
> > > bulk-email-consumer
> > > > [2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > > 869377f9ead1,protocol_type=consumer,group_protocols=[{
> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > > metadata=''}} for group bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25444488 for partition
> > bulk-email-9
> > > > [2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-9] for group
> > > bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > > 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:22] :: DEBUG ::
> > > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > > bulk-email-consumer committed offset 25465113 for partition
> > bulk-email-8
> > > > [2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-8] for group
> > > bulk-email-consumer
> > > > [2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > > 9d083b256ffa,protocol_type=consumer,group_protocols=[{
> > > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:26] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > > [2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0
> > rack:
> > > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:28] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > > [2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2
> > rack:
> > > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:30] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:34] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > > [2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> > rack:
> > > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:45] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 -
> Completed
> > > > autocommit of offsets
> {bulk-email-0=OffsetAndMetadata{offset=25390438,
> > > > metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
> > > > metadata=''},
> > > > bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for
> > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto
> offset
> > > > commit failed for group bulk-email-consumer: Commit cannot be
> completed
> > > > since the group has already rebalanced and assigned the partitions to
> > > > another member. This means that the time between subsequent calls to
> > > poll()
> > > > was longer than the configured max.poll.interval.ms, which typically
> > > > implies that the poll loop is spending too much time message
> > processing.
> > > > You can address this either by increasing the session timeout or by
> > > > reducing the maximum size of batches returned in poll() with
> > > > max.poll.records.
> > > > [2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > > previously assigned partitions [bulk-email-0, bulk-email-2,
> > bulk-email-1]
> > > > for group bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:50] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 0
> > > > [2018-08-09 11:40:50] :: DEBUG ::
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > > LEAVE_GROUP request ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
> > > > request=RequestSend(header={api_key=13,api_version=0,
> > > > correlation_id=349313,client_id=consumer-19},
> > > > body={group_id=bulk-email-consumer,member_id=consumer-
> > > > 19-34903180-9cb3-496d-a913-b782476d9a52}),
> > > > createdTimeMs=1533794419195, sendTimeMs=1533794419195) with
> correlation
> > > id
> > > > 349313 due to node 2147483647 being disconnected
> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: DEBUG ::
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > > JOIN_GROUP request ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
> > > > request=RequestSend(header={api_key=11,api_version=1,
> > > > correlation_id=349314,client_id=consumer-19},
> > > > body={group_id=bulk-email-consumer,session_timeout=
> > > > 300000,rebalance_timeout=300000,member_id=,protocol_
> > > > type=consumer,group_protocols=[{protocol_name=range,
> > > > protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
> > > > correlation id 349314 due to node 2147483647 being disconnected
> > > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking
> the
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > > group
> > > > bulk-email-consumer
> > > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > > coordinator request for group bulk-email-consumer to broker
> > > > 10.189.179.118:9092 (id: 2 rack: null)
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > NetworkClient$DefaultMetadataUpdater:640
> > > > - Sending metadata request {topics=[bulk-email]} to node 2
> > > > [2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster
> > > metadata
> > > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> > rack:
> > > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition =
> 7,
> > > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition =
> 5,
> > > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> > group
> > > > coordinator response ClientResponse(receivedTimeMs=1533795051063,
> > > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > > callback=org.apache.kafka.clients.consumer.internals.
> > > > ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
> > > > request=RequestSend(header={api_key=10,api_version=0,
> > > > correlation_id=349316,client_id=consumer-19},
> > > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
> > > > sendTimeMs=1533795051061),
> > > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > > 10.189.179.116,port=9092}})
> > > > [2018-08-09 11:40:51] :: INFO ::
> > > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for
> group
> > > > bulk-emai -consumer.
> > > > [2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 -
> > (Re-)joining
> > > > group bulk-email-consumer
> > > > [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > > JoinGroup
> > > > ({group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=
> > > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id:
> 2147483647
> > > > rack:
> > > > null)
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:40:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:00] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:17] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:17] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:33] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:36] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:39] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:42] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:46] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:46] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:48] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:51] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:54] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:41:57] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:01] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:03] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:06] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:09] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:12] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:15] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:18] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:21] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:24] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:27] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:31] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:34] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:37] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:40] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:43] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:46] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:49] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:52] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:55] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:42:58] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:01] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:04] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:07] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:10] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:13] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:16] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:19] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:22] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:25] :: DEBUG ::
> > > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> > beat
> > > > failed for group bulk-email-consumer since it is rebalancing.
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG ::
> > > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received
> successful
> > > > join
> > > > group response for group bulk-email-consumer:
> > > > {error_code=0,generation_id=1225,group_protocol=range,
> > > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
> > > > members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > > 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22
> > > > cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > > 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22
> > > > cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > > 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22
> > > > cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > > 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > > lim=22 cap=22]}]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
> > > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 -
> Performing
> > > > assignment for group bulk-email-consumer using strategy range with
> > > > subscriptions
> > > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > > Subscription(topics=[bulk-email]),
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > > Subscription(topics=[bulk-email]),
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > > Subscription(topics=[bulk-email]),
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > > Subscription(topics=[bulk-email])}
> > > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
> > > > assignment for group bulk-email-consumer:
> > > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > > Assignment(partitions=[bulk-email-3,
> > > > bulk-email-4, bulk-email-5]),
> > > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > > Assignment(partitions=[bulk-email-0,
> > > > bulk-email-1, bulk-email-2]),
> > > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > > Assignment(partitions=[bulk-email-6,
> > > > bulk-email-7]),
> > > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > > Assignment(partitions=[bulk-email-8,
> > > > bulk-email-9])}
> > > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending
> > > leader
> > > > SyncGroup for group bulk-email-consumer to coordinator
> > > 10.189.179.116:9092
> > > > (id: 2147483647 rack: null):
> > > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
> > > > assignment=[{member_id=consumer-23-bdc073b6-1f90-
> > > >
> 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > > lim=38
> > > > cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > > 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > > lim=38
> > > > cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > > 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > > lim=34
> > > > cap=34]},{member_id=consumer-26
Re: Very long consumer rebalances
Posted by "M. Manna" <ma...@gmail.com>.
if you can upgrade, I would say upgrading to 0.10.2.x would be better for
you (or even higher, 2.0.0). Otherwise you have to play around with
max.poll.records and session.timeout.ms.
As the doc says (or newer versions), the adjustment should be such that
request.timeout.ms >= max.poll.interval.ms. Also, heartbeat.interval.ms
should be curbed at (rule of thumb) 30% of session.timeout.ms.
Lastly, all these have to be within the bounds of
group.min.session.timeout.ms and group.max.session.timeout.ms.
You can check all these, tune them as necessary, and retry. Some of these
configs may or may not be applicable at runtime. so a rolling restart may
be required before all changes take place.
On 9 August 2018 at 13:48, Shantanu Deshmukh <sh...@gmail.com> wrote:
> Hi,
>
> Yes my consumer application works like below
>
> 1. Reads how many workers are required to process each topics from
> properties file
> 2. As many threads are spawned as there are workers mentioned in
> properties file, topic name is passed to this thread. FixedThreadPool
> implementation is used.
> 3. Each worker thread initializes one consumer object and subscribes to
> given topic. Consumer group is simply <topic-name>-consumer. So if my
> topic
> bulk-email, then consumer group for all those threads is
> bulk-email-consumer
> 4. Once this is done, inside an infinite while loop consumer.poll(100)
> method keeps running. So this application is a daemon. Shuts down only
> when
> server shuts down or in case of kill command.
>
> I have configured session.timeout.ms in consumer properties. I haven't
> done
> anything about zookeeper timeout. Is it required now? Since consumer
> accesses only the brokers.
>
> On Thu, Aug 9, 2018 at 3:03 PM M. Manna <ma...@gmail.com> wrote:
>
> > In the simplest way, how have you implemented your consumer?
> >
> > 1) Does your consumers join a designated group, process messages, and
> then
> > closes all connection? Or does it stay open perpetually until server
> > shutdown?
> > 2) Have you configured the session timeouts for client and zookeeper
> > accordingly?
> >
> > Regards,
> >
> > On 9 August 2018 at 08:00, Shantanu Deshmukh <sh...@gmail.com>
> > wrote:
> >
> > > I am facing too many problems these days. Now one of our consumer
> groups
> > > is rebalancing every now and then. And rebalance takes very low, more
> > than
> > > 5-10 minutes. Even after re-balancing I see that only half of the
> > consumers
> > > are active/receive assignment. Its all going haywire.
> > >
> > > I am seeing these logs in kafka consumer logs. Can anyone help me
> > > understand what is going on here? It is a very long piece of log, but
> > > someone please help me. I am desperately looking for any solution since
> > > more than 2 months now. But to no avail.
> > >
> > > [2018-08-09 11:39:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:53] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:53] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:53] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:56] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:39:56] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:39:56] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:39:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> for
> > > group bulk-email-consumer failed: The request timed out.
> > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> offset
> > > commit failed for group bulk-email-consumer: Offset commit failed with
> a
> > > retriable exception. You should retry committing offsets.
> > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > coordinator request for group bulk-email-consumer to broker
> > > 10.189.179.117:9092 (id: 1 rack: null)
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit
> for
> > > group bulk-email-consumer failed: The request timed out.
> > > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto
> offset
> > > commit failed for group bulk-email-consumer: Offset commit failed with
> a
> > > retriable exception. You should retry committing offsets.
> > > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > coordinator request for group bulk-email-consumer to broker
> > > 10.189.179.118:9092 (id: 2 rack: null)
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> group
> > > coordinator response ClientResponse(receivedTimeMs=1533795004158,
> > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
> > > request=RequestSend(header={api_key=10,api_version=0,
> > > correlation_id=389784,client_id=consumer-26},
> > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
> > > sendTimeMs=1533795004157),
> > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > 10.189.179.116,port=9092}})
> > > [2018-08-09 11:40:04] :: INFO ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > > bulk-emai -consumer.
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> group
> > > coordinator response ClientResponse(receivedTimeMs=1533795004718,
> > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
> > > request=RequestSend(header={api_key=10,api_version=0,
> > > correlation_id=363859,client_id=consumer-25},
> > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
> > > sendTimeMs=1533795004145),
> > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > 10.189.179.116,port=9092}})
> > > [2018-08-09 11:40:04] :: INFO ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > > bulk-emai -consumer.
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:04] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:05] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:07] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:07] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:07] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:07] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:08] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:10] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:10] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:10] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:10] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:11] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:13] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:13] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:14] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:14] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:14] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:16] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:16] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:16] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:16] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:17] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:19] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:19] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:19] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:19] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:40:20] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:21] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > previously assigned partitions [bulk-email-6] for group
> > bulk-email-consumer
> > > [2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > 869377f9ead1,protocol_type=consumer,group_protocols=[{
> > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}} for group bulk-email-consumer
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > previously assigned partitions [bulk-email-9] for group
> > bulk-email-consumer
> > > [2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
> > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:22] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > previously assigned partitions [bulk-email-8] for group
> > bulk-email-consumer
> > > [2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > 9d083b256ffa,protocol_type=consumer,group_protocols=[{
> > > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:26] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > [2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster
> > metadata
> > > version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0
> rack:
> > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:28] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > [2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster
> > metadata
> > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2
> rack:
> > > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > [2018-08-09 11:40:30] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:30] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:30] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:30] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:31] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:34] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 1
> > > [2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster
> > metadata
> > > version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> rack:
> > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:45] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-0=OffsetAndMetadata{offset=25390438,
> > > metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
> > > metadata=''},
> > > bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for
> group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto offset
> > > commit failed for group bulk-email-consumer: Commit cannot be completed
> > > since the group has already rebalanced and assigned the partitions to
> > > another member. This means that the time between subsequent calls to
> > poll()
> > > was longer than the configured max.poll.interval.ms, which typically
> > > implies that the poll loop is spending too much time message
> processing.
> > > You can address this either by increasing the session timeout or by
> > > reducing the maximum size of batches returned in poll() with
> > > max.poll.records.
> > > [2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
> > > previously assigned partitions [bulk-email-0, bulk-email-2,
> bulk-email-1]
> > > for group bulk-email-consumer
> > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:50] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 0
> > > [2018-08-09 11:40:50] :: DEBUG ::
> > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > LEAVE_GROUP request ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
> > > request=RequestSend(header={api_key=13,api_version=0,
> > > correlation_id=349313,client_id=consumer-19},
> > > body={group_id=bulk-email-consumer,member_id=consumer-
> > > 19-34903180-9cb3-496d-a913-b782476d9a52}),
> > > createdTimeMs=1533794419195, sendTimeMs=1533794419195) with correlation
> > id
> > > 349313 due to node 2147483647 being disconnected
> > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:50] :: DEBUG ::
> > > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > > JOIN_GROUP request ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
> > > request=RequestSend(header={api_key=11,api_version=1,
> > > correlation_id=349314,client_id=consumer-19},
> > > body={group_id=bulk-email-consumer,session_timeout=
> > > 300000,rebalance_timeout=300000,member_id=,protocol_
> > > type=consumer,group_protocols=[{protocol_name=range,
> > > protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
> > > correlation id 349314 due to node 2147483647 being disconnected
> > > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> > group
> > > bulk-email-consumer
> > > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
> > > coordinator request for group bulk-email-consumer to broker
> > > 10.189.179.118:9092 (id: 2 rack: null)
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > NetworkClient$DefaultMetadataUpdater:640
> > > - Sending metadata request {topics=[bulk-email]} to node 2
> > > [2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster
> > metadata
> > > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0
> rack:
> > > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received
> group
> > > coordinator response ClientResponse(receivedTimeMs=1533795051063,
> > > disconnected=false, request=ClientRequest(expectResponse=true,
> > > callback=org.apache.kafka.clients.consumer.internals.
> > > ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
> > > request=RequestSend(header={api_key=10,api_version=0,
> > > correlation_id=349316,client_id=consumer-19},
> > > body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
> > > sendTimeMs=1533795051061),
> > > responseBody={error_code=0,coordinator={node_id=0,host=
> > > 10.189.179.116,port=9092}})
> > > [2018-08-09 11:40:51] :: INFO ::
> > > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > > bulk-emai -consumer.
> > > [2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 -
> (Re-)joining
> > > group bulk-email-consumer
> > > [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
> > > JoinGroup
> > > ({group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=
> > > 300000,member_id=,protocol_type=consumer,group_protocols=
> > > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > > rack:
> > > null)
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:40:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:00] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:17] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:17] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:31] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:31] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:33] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:36] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:39] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:42] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:46] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:46] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:48] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:51] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:54] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:41:57] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:01] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:03] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:06] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:09] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:12] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:15] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:18] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:21] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:24] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:27] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:31] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:34] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:37] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:40] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:43] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:46] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:49] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:52] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:55] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:42:58] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:01] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:04] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:07] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:10] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:13] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:16] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:19] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:22] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:25] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart
> beat
> > > failed for group bulk-email-consumer since it is rebalancing.
> > > [2018-08-09 11:43:26] :: DEBUG ::
> > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > > join
> > > group response for group bulk-email-consumer:
> > > {error_code=0,generation_id=1225,group_protocol=range,
> > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
> > > [2018-08-09 11:43:26] :: DEBUG ::
> > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > > join
> > > group response for group bulk-email-consumer:
> > > {error_code=0,generation_id=1225,group_protocol=range,
> > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
> > > [2018-08-09 11:43:26] :: DEBUG ::
> > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > > join
> > > group response for group bulk-email-consumer:
> > > {error_code=0,generation_id=1225,group_protocol=range,
> > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
> > > [2018-08-09 11:43:26] :: DEBUG ::
> > > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > > join
> > > group response for group bulk-email-consumer:
> > > {error_code=0,generation_id=1225,group_protocol=range,
> > > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
> > > members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > > 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22
> > > cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22
> > > cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22
> > > cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
> > > lim=22 cap=22]}]}
> > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
> > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
> > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > > follower SyncGroup for group bulk-email-consumer to coordinator
> > > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 - Performing
> > > assignment for group bulk-email-consumer using strategy range with
> > > subscriptions
> > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > Subscription(topics=[bulk-email]),
> > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > Subscription(topics=[bulk-email]),
> > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > Subscription(topics=[bulk-email]),
> > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > Subscription(topics=[bulk-email])}
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
> > > assignment for group bulk-email-consumer:
> > > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > > Assignment(partitions=[bulk-email-3,
> > > bulk-email-4, bulk-email-5]),
> > > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > > Assignment(partitions=[bulk-email-0,
> > > bulk-email-1, bulk-email-2]),
> > > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > > Assignment(partitions=[bulk-email-6,
> > > bulk-email-7]),
> > > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > > Assignment(partitions=[bulk-email-8,
> > > bulk-email-9])}
> > > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending
> > leader
> > > SyncGroup for group bulk-email-consumer to coordinator
> > 10.189.179.116:9092
> > > (id: 2147483647 rack: null):
> > > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
> > > assignment=[{member_id=consumer-23-bdc073b6-1f90-
> > > 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > lim=38
> > > cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > > 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > lim=38
> > > cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > > 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > lim=34
> > > cap=34]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > > 1c4dd363ba09,member_assignment=java.nio.HeapByteBuffer[pos=0
> > > lim=34 cap=34]}]}
> > > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> > Successfully
> > > joined group bulk-email-consumer with generation 1225
> > > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> > Successfully
> > > joined group bulk-email-consumer with generation 1225
> > > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> > Successfully
> > > joined group bulk-email-consumer with generation 1225
> > > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> > Successfully
> > > joined group bulk-email-consumer with generation 1225
> > > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting
> newly
> > > assigned partitions [bulk-email-7, bulk-email-6] for group
> > > bulk-email-consumer
> > > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting
> newly
> > > assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1] for
> group
> > > bulk-email-consumer
> > > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting
> newly
> > > assigned partitions [bulk-email-4, bulk-email-3, bulk-email-5] for
> group
> > > bulk-email-consumer
> > > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting
> newly
> > > assigned partitions [bulk-email-8, bulk-email-9] for group
> > > bulk-email-consumer
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > > bulk-email-consumer fetching committed offsets for partitions:
> > > [bulk-email-7, bulk-email-6]
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > > bulk-email-consumer fetching committed offsets for partitions:
> > > [bulk-email-0, bulk-email-2, bulk-email-1]
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > > bulk-email-consumer fetching committed offsets for partitions:
> > > [bulk-email-4, bulk-email-3, bulk-email-5]
> > > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > > bulk-email-consumer fetching committed offsets for partitions:
> > > [bulk-email-8, bulk-email-9]
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-7 to the committed offset 25435998
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-0 to the committed offset 25390842
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-6 to the committed offset 25463566
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-4 to the committed offset 25409183
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-8 to the committed offset 25465113
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-2 to the committed offset 25368327
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched
> > records
> > > for partition bulk-email-8 since it is no longer fetchable
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-3 to the committed offset 25422008
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-9 to the committed offset 25444488
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-1 to the committed offset 25369263
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > > partition bulk-email-5 to the committed offset 25456518
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :470 - Ignoring fetched
> > records
> > > for bulk-email-0 at offset 25390442 since the current position is
> > 25390842
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched
> > records
> > > for partition bulk-email-6 since it is no longer fetchable
> > > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :671 - Discarding stale
> fetch
> > > response for partition bulk-email-1 since its offset 25369219 does not
> > > match the expected offset 25369263
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25465113 for partition
> bulk-email-8
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25444488 for partition
> bulk-email-9
> > > [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > > metadata=''}, bulk-email-9=OffsetAndMetadata{offset=25444488,
> > > metadata=''}}
> > > for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25436018 for partition
> bulk-email-7
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25463566 for partition
> bulk-email-6
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > > heartbeat response for group bulk-email-consumer
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25409183 for partition
> bulk-email-4
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25422008 for partition
> bulk-email-3
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25456530 for partition
> bulk-email-5
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25390842 for partition
> bulk-email-0
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25368339 for partition
> bulk-email-2
> > > [2018-08-09 11:43:29] :: DEBUG ::
> > > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > > bulk-email-consumer committed offset 25369263 for partition
> bulk-email-1
> > > [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > > autocommit of offsets {bulk-email-4=OffsetAndMetadata{offset=25409183,
> > > metadata=''}, bulk-email-3=OffsetAndMetadata{offset=25422008,
> > > metadata=''},
> > > bulk-email-5=OffsetAndMetadata{offset=25456530, metadata=''}} for
> group
> > > bulk-email-consumer
> > >
> > > On Thu, Jul 12, 2018 at 9:27 PM Steve Tian <st...@gmail.com>
> > > wrote:
> > >
> > > > It's a very good and important doc so I think you should read it all.
> > > You
> > > > should get some idea from sections like *Detecting Consumer Failures*
> > and
> > > > *Multi-threaded Processing* for your case.
> > > >
> > > > On Thu, Jul 12, 2018, 3:17 PM Shantanu Deshmukh <
> shantanu88d@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Steve,
> > > > >
> > > > > Could you please shed more light on this? What section should I
> > > revisit?
> > > > I
> > > > > am using high-level consumer. So I am simply calling
> consumer.close()
> > > > when
> > > > > I am shutting down the process. Is there any other method to be
> > called
> > > > > before calling close()?
> > > > >
> > > > > On Mon, Jul 9, 2018 at 5:58 PM Steve Tian <steve.cs.tian@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Please re-read the javadoc of KafkaConsumer, make sure you know
> how
> > > to
> > > > > > wakeup/close consumer properly while shutting down your
> > application.
> > > > Try
> > > > > > to understand the motivation of KIP-62 and adjust related
> timeout.
> > > > > >
> > > > > > On Mon, Jul 9, 2018, 8:05 PM harish lohar <hk...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Try reducing below timer
> > > > > > > metadata.max.age.ms = 300000
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 6, 2018 at 5:55 AM Shantanu Deshmukh <
> > > > > shantanu88d@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello everyone,
> > > > > > > >
> > > > > > > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a
> > java
> > > > app
> > > > > > > which
> > > > > > > > spawns many consumer threads consuming from different topics.
> > For
> > > > > every
> > > > > > > > topic we have specified different consumer-group. A lot of
> > times
> > > I
> > > > > see
> > > > > > > that
> > > > > > > > whenever this application is restarted a CG on one or two
> > topics
> > > > > takes
> > > > > > > more
> > > > > > > > than 5 minutes to receive partition assignment. Till that
> time
> > > > > > consumers
> > > > > > > > for that topic don't consumer anything. If I go to Kafka
> broker
> > > and
> > > > > run
> > > > > > > > consumer-groups.sh and describe that particular CG I see that
> > it
> > > is
> > > > > > > > rebalancing. There is time critical data stored in that topic
> > and
> > > > we
> > > > > > > cannot
> > > > > > > > tolerate such long delays. What can be the reason for such
> long
> > > > > > > rebalances.
> > > > > > > >
> > > > > > > > Here's our consumer config
> > > > > > > >
> > > > > > > >
> > > > > > > > auto.commit.interval.ms = 3000
> > > > > > > > auto.offset.reset = latest
> > > > > > > > bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092,
> x.x.x.x:9092]
> > > > > > > > check.crcs = true
> > > > > > > > client.id =
> > > > > > > > connections.max.idle.ms = 540000
> > > > > > > > enable.auto.commit = true
> > > > > > > > exclude.internal.topics = true
> > > > > > > > fetch.max.bytes = 52428800
> > > > > > > > fetch.max.wait.ms = 500
> > > > > > > > fetch.min.bytes = 1
> > > > > > > > group.id = otp-notifications-consumer
> > > > > > > > heartbeat.interval.ms = 3000
> > > > > > > > interceptor.classes = null
> > > > > > > > key.deserializer = class
> > > > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > > > max.partition.fetch.bytes = 1048576
> > > > > > > > max.poll.interval.ms = 300000
> > > > > > > > max.poll.records = 50
> > > > > > > > metadata.max.age.ms = 300000
> > > > > > > > metric.reporters = []
> > > > > > > > metrics.num.samples = 2
> > > > > > > > metrics.sample.window.ms = 30000
> > > > > > > > partition.assignment.strategy = [class
> > > > > > > > org.apache.kafka.clients.consumer.RangeAssignor]
> > > > > > > > receive.buffer.bytes = 65536
> > > > > > > > reconnect.backoff.ms = 50
> > > > > > > > request.timeout.ms = 305000
> > > > > > > > retry.backoff.ms = 100
> > > > > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit
> > > > > > > > sasl.kerberos.min.time.before.relogin = 60000
> > > > > > > > sasl.kerberos.service.name = null
> > > > > > > > sasl.kerberos.ticket.renew.jitter = 0.05
> > > > > > > > sasl.kerberos.ticket.renew.window.factor = 0.8
> > > > > > > > sasl.mechanism = GSSAPI
> > > > > > > > security.protocol = SSL
> > > > > > > > send.buffer.bytes = 131072
> > > > > > > > session.timeout.ms = 300000
> > > > > > > > ssl.cipher.suites = null
> > > > > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> > > > > > > > ssl.endpoint.identification.algorithm = null
> > > > > > > > ssl.key.password = null
> > > > > > > > ssl.keymanager.algorithm = SunX509
> > > > > > > > ssl.keystore.location = null
> > > > > > > > ssl.keystore.password = null
> > > > > > > > ssl.keystore.type = JKS
> > > > > > > > ssl.protocol = TLS
> > > > > > > > ssl.provider = null
> > > > > > > > ssl.secure.random.implementation = null
> > > > > > > > ssl.trustmanager.algorithm = PKIX
> > > > > > > > ssl.truststore.location = /x/x/client.truststore.jks
> > > > > > > > ssl.truststore.password = [hidden]
> > > > > > > > ssl.truststore.type = JKS
> > > > > > > > value.deserializer = class
> > > > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > > >
> > > > > > > > Please help.
> > > > > > > >
> > > > > > > > *Thanks & Regards,*
> > > > > > > > *Shantanu Deshmukh*
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Very long consumer rebalances
Posted by Shantanu Deshmukh <sh...@gmail.com>.
Hi,
Yes my consumer application works like below
1. Reads how many workers are required to process each topics from
properties file
2. As many threads are spawned as there are workers mentioned in
properties file, topic name is passed to this thread. FixedThreadPool
implementation is used.
3. Each worker thread initializes one consumer object and subscribes to
given topic. Consumer group is simply <topic-name>-consumer. So if my topic
bulk-email, then consumer group for all those threads is bulk-email-consumer
4. Once this is done, inside an infinite while loop consumer.poll(100)
method keeps running. So this application is a daemon. Shuts down only when
server shuts down or in case of kill command.
I have configured session.timeout.ms in consumer properties. I haven't done
anything about zookeeper timeout. Is it required now? Since consumer
accesses only the brokers.
On Thu, Aug 9, 2018 at 3:03 PM M. Manna <ma...@gmail.com> wrote:
> In the simplest way, how have you implemented your consumer?
>
> 1) Does your consumers join a designated group, process messages, and then
> closes all connection? Or does it stay open perpetually until server
> shutdown?
> 2) Have you configured the session timeouts for client and zookeeper
> accordingly?
>
> Regards,
>
> On 9 August 2018 at 08:00, Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
> > I am facing too many problems these days. Now one of our consumer groups
> > is rebalancing every now and then. And rebalance takes very low, more
> than
> > 5-10 minutes. Even after re-balancing I see that only half of the
> consumers
> > are active/receive assignment. Its all going haywire.
> >
> > I am seeing these logs in kafka consumer logs. Can anyone help me
> > understand what is going on here? It is a very long piece of log, but
> > someone please help me. I am desperately looking for any solution since
> > more than 2 months now. But to no avail.
> >
> > [2018-08-09 11:39:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
> > group bulk-email-consumer failed: The request timed out.
> > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> group
> > bulk-email-consumer
> > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
> > commit failed for group bulk-email-consumer: Offset commit failed with a
> > retriable exception. You should retry committing offsets.
> > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > coordinator request for group bulk-email-consumer to broker
> > 10.189.179.117:9092 (id: 1 rack: null)
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
> > group bulk-email-consumer failed: The request timed out.
> > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> group
> > bulk-email-consumer
> > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
> > commit failed for group bulk-email-consumer: Offset commit failed with a
> > retriable exception. You should retry committing offsets.
> > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > coordinator request for group bulk-email-consumer to broker
> > 10.189.179.118:9092 (id: 2 rack: null)
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> > coordinator response ClientResponse(receivedTimeMs=1533795004158,
> > disconnected=false, request=ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
> > request=RequestSend(header={api_key=10,api_version=0,
> > correlation_id=389784,client_id=consumer-26},
> > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
> > sendTimeMs=1533795004157),
> > responseBody={error_code=0,coordinator={node_id=0,host=
> > 10.189.179.116,port=9092}})
> > [2018-08-09 11:40:04] :: INFO ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > bulk-emai -consumer.
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> > coordinator response ClientResponse(receivedTimeMs=1533795004718,
> > disconnected=false, request=ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
> > request=RequestSend(header={api_key=10,api_version=0,
> > correlation_id=363859,client_id=consumer-25},
> > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
> > sendTimeMs=1533795004145),
> > responseBody={error_code=0,coordinator={node_id=0,host=
> > 10.189.179.116,port=9092}})
> > [2018-08-09 11:40:04] :: INFO ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > bulk-emai -consumer.
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:05] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:07] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:07] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:07] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:07] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:08] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:10] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:10] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:10] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:10] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:11] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:13] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:13] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:14] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:14] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:14] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:16] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:16] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:16] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:16] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:17] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:19] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:19] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:19] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:19] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:20] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
> > previously assigned partitions [bulk-email-6] for group
> bulk-email-consumer
> > [2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > 869377f9ead1,protocol_type=consumer,group_protocols=[{
> > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:22] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:22] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:22] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:22] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:22] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
> > previously assigned partitions [bulk-email-9] for group
> bulk-email-consumer
> > [2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
> > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:22] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
> > previously assigned partitions [bulk-email-8] for group
> bulk-email-consumer
> > [2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > 9d083b256ffa,protocol_type=consumer,group_protocols=[{
> > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:26] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 1
> > [2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster
> metadata
> > version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0 rack:
> > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:28] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 1
> > [2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster
> metadata
> > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2 rack:
> > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > [2018-08-09 11:40:30] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:30] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:30] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:30] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:31] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:34] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 1
> > [2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster
> metadata
> > version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
> > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-0=OffsetAndMetadata{offset=25390438,
> > metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
> > metadata=''},
> > bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for group
> > bulk-email-consumer
> > [2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto offset
> > commit failed for group bulk-email-consumer: Commit cannot be completed
> > since the group has already rebalanced and assigned the partitions to
> > another member. This means that the time between subsequent calls to
> poll()
> > was longer than the configured max.poll.interval.ms, which typically
> > implies that the poll loop is spending too much time message processing.
> > You can address this either by increasing the session timeout or by
> > reducing the maximum size of batches returned in poll() with
> > max.poll.records.
> > [2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
> > previously assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1]
> > for group bulk-email-consumer
> > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=,protocol_type=consumer,group_protocols=
> > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:50] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 0
> > [2018-08-09 11:40:50] :: DEBUG ::
> > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > LEAVE_GROUP request ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
> > request=RequestSend(header={api_key=13,api_version=0,
> > correlation_id=349313,client_id=consumer-19},
> > body={group_id=bulk-email-consumer,member_id=consumer-
> > 19-34903180-9cb3-496d-a913-b782476d9a52}),
> > createdTimeMs=1533794419195, sendTimeMs=1533794419195) with correlation
> id
> > 349313 due to node 2147483647 being disconnected
> > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> group
> > bulk-email-consumer
> > [2018-08-09 11:40:50] :: DEBUG ::
> > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > JOIN_GROUP request ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
> > request=RequestSend(header={api_key=11,api_version=1,
> > correlation_id=349314,client_id=consumer-19},
> > body={group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=300000,member_id=,protocol_
> > type=consumer,group_protocols=[{protocol_name=range,
> > protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
> > correlation id 349314 due to node 2147483647 being disconnected
> > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> group
> > bulk-email-consumer
> > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
> > coordinator request for group bulk-email-consumer to broker
> > 10.189.179.118:9092 (id: 2 rack: null)
> > [2018-08-09 11:40:51] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 2
> > [2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster
> metadata
> > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
> > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> > coordinator response ClientResponse(receivedTimeMs=1533795051063,
> > disconnected=false, request=ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
> > request=RequestSend(header={api_key=10,api_version=0,
> > correlation_id=349316,client_id=consumer-19},
> > body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
> > sendTimeMs=1533795051061),
> > responseBody={error_code=0,coordinator={node_id=0,host=
> > 10.189.179.116,port=9092}})
> > [2018-08-09 11:40:51] :: INFO ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > bulk-emai -consumer.
> > [2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=,protocol_type=consumer,group_protocols=
> > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:17] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:17] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:31] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:31] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:46] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:46] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:01] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:31] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:34] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:37] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:40] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:43] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:46] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:49] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:52] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:55] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:58] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:01] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:07] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:10] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:13] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:16] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:19] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:22] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:25] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:26] :: DEBUG ::
> > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > join
> > group response for group bulk-email-consumer:
> > {error_code=0,generation_id=1225,group_protocol=range,
> > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
> > [2018-08-09 11:43:26] :: DEBUG ::
> > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > join
> > group response for group bulk-email-consumer:
> > {error_code=0,generation_id=1225,group_protocol=range,
> > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
> > [2018-08-09 11:43:26] :: DEBUG ::
> > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > join
> > group response for group bulk-email-consumer:
> > {error_code=0,generation_id=1225,group_protocol=range,
> > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
> > [2018-08-09 11:43:26] :: DEBUG ::
> > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > join
> > group response for group bulk-email-consumer:
> > {error_code=0,generation_id=1225,group_protocol=range,
> > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
> > members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22
> > cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22
> > cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22
> > cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}
> > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > follower SyncGroup for group bulk-email-consumer to coordinator
> > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
> > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > follower SyncGroup for group bulk-email-consumer to coordinator
> > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
> > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > follower SyncGroup for group bulk-email-consumer to coordinator
> > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 - Performing
> > assignment for group bulk-email-consumer using strategy range with
> > subscriptions
> > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > Subscription(topics=[bulk-email]),
> > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > Subscription(topics=[bulk-email]),
> > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > Subscription(topics=[bulk-email]),
> > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > Subscription(topics=[bulk-email])}
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
> > assignment for group bulk-email-consumer:
> > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > Assignment(partitions=[bulk-email-3,
> > bulk-email-4, bulk-email-5]),
> > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > Assignment(partitions=[bulk-email-0,
> > bulk-email-1, bulk-email-2]),
> > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > Assignment(partitions=[bulk-email-6,
> > bulk-email-7]),
> > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > Assignment(partitions=[bulk-email-8,
> > bulk-email-9])}
> > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending
> leader
> > SyncGroup for group bulk-email-consumer to coordinator
> 10.189.179.116:9092
> > (id: 2147483647 rack: null):
> > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
> > assignment=[{member_id=consumer-23-bdc073b6-1f90-
> > 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
> > lim=38
> > cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > lim=38
> > cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > lim=34
> > cap=34]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > 1c4dd363ba09,member_assignment=java.nio.HeapByteBuffer[pos=0
> > lim=34 cap=34]}]}
> > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> Successfully
> > joined group bulk-email-consumer with generation 1225
> > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> Successfully
> > joined group bulk-email-consumer with generation 1225
> > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> Successfully
> > joined group bulk-email-consumer with generation 1225
> > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> Successfully
> > joined group bulk-email-consumer with generation 1225
> > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> > assigned partitions [bulk-email-7, bulk-email-6] for group
> > bulk-email-consumer
> > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> > assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1] for group
> > bulk-email-consumer
> > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> > assigned partitions [bulk-email-4, bulk-email-3, bulk-email-5] for group
> > bulk-email-consumer
> > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> > assigned partitions [bulk-email-8, bulk-email-9] for group
> > bulk-email-consumer
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > bulk-email-consumer fetching committed offsets for partitions:
> > [bulk-email-7, bulk-email-6]
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > bulk-email-consumer fetching committed offsets for partitions:
> > [bulk-email-0, bulk-email-2, bulk-email-1]
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > bulk-email-consumer fetching committed offsets for partitions:
> > [bulk-email-4, bulk-email-3, bulk-email-5]
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > bulk-email-consumer fetching committed offsets for partitions:
> > [bulk-email-8, bulk-email-9]
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-7 to the committed offset 25435998
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-0 to the committed offset 25390842
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-6 to the committed offset 25463566
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-4 to the committed offset 25409183
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-8 to the committed offset 25465113
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-2 to the committed offset 25368327
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched
> records
> > for partition bulk-email-8 since it is no longer fetchable
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-3 to the committed offset 25422008
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-9 to the committed offset 25444488
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-1 to the committed offset 25369263
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-5 to the committed offset 25456518
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :470 - Ignoring fetched
> records
> > for bulk-email-0 at offset 25390442 since the current position is
> 25390842
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched
> records
> > for partition bulk-email-6 since it is no longer fetchable
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :671 - Discarding stale fetch
> > response for partition bulk-email-1 since its offset 25369219 does not
> > match the expected offset 25369263
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}, bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}}
> > for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25436018 for partition bulk-email-7
> > [2018-08-09 11:43:29] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:43:29] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25409183 for partition bulk-email-4
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25422008 for partition bulk-email-3
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25456530 for partition bulk-email-5
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25390842 for partition bulk-email-0
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25368339 for partition bulk-email-2
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25369263 for partition bulk-email-1
> > [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-4=OffsetAndMetadata{offset=25409183,
> > metadata=''}, bulk-email-3=OffsetAndMetadata{offset=25422008,
> > metadata=''},
> > bulk-email-5=OffsetAndMetadata{offset=25456530, metadata=''}} for group
> > bulk-email-consumer
> >
> > On Thu, Jul 12, 2018 at 9:27 PM Steve Tian <st...@gmail.com>
> > wrote:
> >
> > > It's a very good and important doc so I think you should read it all.
> > You
> > > should get some idea from sections like *Detecting Consumer Failures*
> and
> > > *Multi-threaded Processing* for your case.
> > >
> > > On Thu, Jul 12, 2018, 3:17 PM Shantanu Deshmukh <shantanu88d@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Steve,
> > > >
> > > > Could you please shed more light on this? What section should I
> > revisit?
> > > I
> > > > am using high-level consumer. So I am simply calling consumer.close()
> > > when
> > > > I am shutting down the process. Is there any other method to be
> called
> > > > before calling close()?
> > > >
> > > > On Mon, Jul 9, 2018 at 5:58 PM Steve Tian <st...@gmail.com>
> > > wrote:
> > > >
> > > > > Please re-read the javadoc of KafkaConsumer, make sure you know how
> > to
> > > > > wakeup/close consumer properly while shutting down your
> application.
> > > Try
> > > > > to understand the motivation of KIP-62 and adjust related timeout.
> > > > >
> > > > > On Mon, Jul 9, 2018, 8:05 PM harish lohar <hk...@gmail.com>
> wrote:
> > > > >
> > > > > > Try reducing below timer
> > > > > > metadata.max.age.ms = 300000
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 6, 2018 at 5:55 AM Shantanu Deshmukh <
> > > > shantanu88d@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a
> java
> > > app
> > > > > > which
> > > > > > > spawns many consumer threads consuming from different topics.
> For
> > > > every
> > > > > > > topic we have specified different consumer-group. A lot of
> times
> > I
> > > > see
> > > > > > that
> > > > > > > whenever this application is restarted a CG on one or two
> topics
> > > > takes
> > > > > > more
> > > > > > > than 5 minutes to receive partition assignment. Till that time
> > > > > consumers
> > > > > > > for that topic don't consumer anything. If I go to Kafka broker
> > and
> > > > run
> > > > > > > consumer-groups.sh and describe that particular CG I see that
> it
> > is
> > > > > > > rebalancing. There is time critical data stored in that topic
> and
> > > we
> > > > > > cannot
> > > > > > > tolerate such long delays. What can be the reason for such long
> > > > > > rebalances.
> > > > > > >
> > > > > > > Here's our consumer config
> > > > > > >
> > > > > > >
> > > > > > > auto.commit.interval.ms = 3000
> > > > > > > auto.offset.reset = latest
> > > > > > > bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092, x.x.x.x:9092]
> > > > > > > check.crcs = true
> > > > > > > client.id =
> > > > > > > connections.max.idle.ms = 540000
> > > > > > > enable.auto.commit = true
> > > > > > > exclude.internal.topics = true
> > > > > > > fetch.max.bytes = 52428800
> > > > > > > fetch.max.wait.ms = 500
> > > > > > > fetch.min.bytes = 1
> > > > > > > group.id = otp-notifications-consumer
> > > > > > > heartbeat.interval.ms = 3000
> > > > > > > interceptor.classes = null
> > > > > > > key.deserializer = class
> > > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > > max.partition.fetch.bytes = 1048576
> > > > > > > max.poll.interval.ms = 300000
> > > > > > > max.poll.records = 50
> > > > > > > metadata.max.age.ms = 300000
> > > > > > > metric.reporters = []
> > > > > > > metrics.num.samples = 2
> > > > > > > metrics.sample.window.ms = 30000
> > > > > > > partition.assignment.strategy = [class
> > > > > > > org.apache.kafka.clients.consumer.RangeAssignor]
> > > > > > > receive.buffer.bytes = 65536
> > > > > > > reconnect.backoff.ms = 50
> > > > > > > request.timeout.ms = 305000
> > > > > > > retry.backoff.ms = 100
> > > > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit
> > > > > > > sasl.kerberos.min.time.before.relogin = 60000
> > > > > > > sasl.kerberos.service.name = null
> > > > > > > sasl.kerberos.ticket.renew.jitter = 0.05
> > > > > > > sasl.kerberos.ticket.renew.window.factor = 0.8
> > > > > > > sasl.mechanism = GSSAPI
> > > > > > > security.protocol = SSL
> > > > > > > send.buffer.bytes = 131072
> > > > > > > session.timeout.ms = 300000
> > > > > > > ssl.cipher.suites = null
> > > > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> > > > > > > ssl.endpoint.identification.algorithm = null
> > > > > > > ssl.key.password = null
> > > > > > > ssl.keymanager.algorithm = SunX509
> > > > > > > ssl.keystore.location = null
> > > > > > > ssl.keystore.password = null
> > > > > > > ssl.keystore.type = JKS
> > > > > > > ssl.protocol = TLS
> > > > > > > ssl.provider = null
> > > > > > > ssl.secure.random.implementation = null
> > > > > > > ssl.trustmanager.algorithm = PKIX
> > > > > > > ssl.truststore.location = /x/x/client.truststore.jks
> > > > > > > ssl.truststore.password = [hidden]
> > > > > > > ssl.truststore.type = JKS
> > > > > > > value.deserializer = class
> > > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > >
> > > > > > > Please help.
> > > > > > >
> > > > > > > *Thanks & Regards,*
> > > > > > > *Shantanu Deshmukh*
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Very long consumer rebalances
Posted by Kamal Chandraprakash <ka...@gmail.com>.
In v0.10.0.1, consumer heartbeat background thread feature is not
available.
Lot of users faced similar errors. So, KIP-62
<https://cwiki-test.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread>
is
proposed. You have to update your
Kafka version.
I highly recommend you to upgrade Kafka to the version where the heartbeat
background
thread feature is implemented (v0.10.1.0). If you don't have any option to
upgrade, you
have to heartbeat the co-ordinator manually from your consumer. You can use
this
<https://github.com/omkreddy/kafka-examples/blob/master/consumer/src/main/java/kafka/examples/consumer/advanced/AdvancedConsumer.java>
code
snippet for reference.
On Thu, Aug 9, 2018 at 3:03 PM M. Manna <ma...@gmail.com> wrote:
> In the simplest way, how have you implemented your consumer?
>
> 1) Does your consumers join a designated group, process messages, and then
> closes all connection? Or does it stay open perpetually until server
> shutdown?
> 2) Have you configured the session timeouts for client and zookeeper
> accordingly?
>
> Regards,
>
> On 9 August 2018 at 08:00, Shantanu Deshmukh <sh...@gmail.com>
> wrote:
>
> > I am facing too many problems these days. Now one of our consumer groups
> > is rebalancing every now and then. And rebalance takes very low, more
> than
> > 5-10 minutes. Even after re-balancing I see that only half of the
> consumers
> > are active/receive assignment. Its all going haywire.
> >
> > I am seeing these logs in kafka consumer logs. Can anyone help me
> > understand what is going on here? It is a very long piece of log, but
> > someone please help me. I am desperately looking for any solution since
> > more than 2 months now. But to no avail.
> >
> > [2018-08-09 11:39:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:53] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:39:56] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:39:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
> > group bulk-email-consumer failed: The request timed out.
> > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> group
> > bulk-email-consumer
> > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
> > commit failed for group bulk-email-consumer: Offset commit failed with a
> > retriable exception. You should retry committing offsets.
> > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > coordinator request for group bulk-email-consumer to broker
> > 10.189.179.117:9092 (id: 1 rack: null)
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
> > group bulk-email-consumer failed: The request timed out.
> > [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> group
> > bulk-email-consumer
> > [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
> > commit failed for group bulk-email-consumer: Offset commit failed with a
> > retriable exception. You should retry committing offsets.
> > [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> > coordinator request for group bulk-email-consumer to broker
> > 10.189.179.118:9092 (id: 2 rack: null)
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> > coordinator response ClientResponse(receivedTimeMs=1533795004158,
> > disconnected=false, request=ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
> > request=RequestSend(header={api_key=10,api_version=0,
> > correlation_id=389784,client_id=consumer-26},
> > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
> > sendTimeMs=1533795004157),
> > responseBody={error_code=0,coordinator={node_id=0,host=
> > 10.189.179.116,port=9092}})
> > [2018-08-09 11:40:04] :: INFO ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > bulk-emai -consumer.
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> > coordinator response ClientResponse(receivedTimeMs=1533795004718,
> > disconnected=false, request=ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
> > request=RequestSend(header={api_key=10,api_version=0,
> > correlation_id=363859,client_id=consumer-25},
> > body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
> > sendTimeMs=1533795004145),
> > responseBody={error_code=0,coordinator={node_id=0,host=
> > 10.189.179.116,port=9092}})
> > [2018-08-09 11:40:04] :: INFO ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > bulk-emai -consumer.
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:04] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:05] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:07] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:07] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:07] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:07] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:08] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:10] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:10] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:10] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:10] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:11] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:13] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:13] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:14] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:14] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:14] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:16] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:16] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:16] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:16] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:17] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:19] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:19] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:19] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:19] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:40:20] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:21] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
> > previously assigned partitions [bulk-email-6] for group
> bulk-email-consumer
> > [2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > 869377f9ead1,protocol_type=consumer,group_protocols=[{
> > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:22] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:22] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}} for group bulk-email-consumer
> > [2018-08-09 11:40:22] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:22] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:22] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
> > previously assigned partitions [bulk-email-9] for group
> bulk-email-consumer
> > [2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
> > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:22] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
> > previously assigned partitions [bulk-email-8] for group
> bulk-email-consumer
> > [2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > 9d083b256ffa,protocol_type=consumer,group_protocols=[{
> > protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:26] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 1
> > [2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster
> metadata
> > version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0 rack:
> > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:28] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 1
> > [2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster
> metadata
> > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2 rack:
> > null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > [2018-08-09 11:40:30] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:30] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:30] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:30] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:31] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:34] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 1
> > [2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster
> metadata
> > version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
> > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:45] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-0=OffsetAndMetadata{offset=25390438,
> > metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
> > metadata=''},
> > bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for group
> > bulk-email-consumer
> > [2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto offset
> > commit failed for group bulk-email-consumer: Commit cannot be completed
> > since the group has already rebalanced and assigned the partitions to
> > another member. This means that the time between subsequent calls to
> poll()
> > was longer than the configured max.poll.interval.ms, which typically
> > implies that the poll loop is spending too much time message processing.
> > You can address this either by increasing the session timeout or by
> > reducing the maximum size of batches returned in poll() with
> > max.poll.records.
> > [2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
> > previously assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1]
> > for group bulk-email-consumer
> > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=,protocol_type=consumer,group_protocols=
> > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:50] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 0
> > [2018-08-09 11:40:50] :: DEBUG ::
> > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > LEAVE_GROUP request ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
> > request=RequestSend(header={api_key=13,api_version=0,
> > correlation_id=349313,client_id=consumer-19},
> > body={group_id=bulk-email-consumer,member_id=consumer-
> > 19-34903180-9cb3-496d-a913-b782476d9a52}),
> > createdTimeMs=1533794419195, sendTimeMs=1533794419195) with correlation
> id
> > 349313 due to node 2147483647 being disconnected
> > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> group
> > bulk-email-consumer
> > [2018-08-09 11:40:50] :: DEBUG ::
> > ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> > JOIN_GROUP request ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
> > request=RequestSend(header={api_key=11,api_version=1,
> > correlation_id=349314,client_id=consumer-19},
> > body={group_id=bulk-email-consumer,session_timeout=
> > 300000,rebalance_timeout=300000,member_id=,protocol_
> > type=consumer,group_protocols=[{protocol_name=range,
> > protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
> > correlation id 349314 due to node 2147483647 being disconnected
> > [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for
> group
> > bulk-email-consumer
> > [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
> > coordinator request for group bulk-email-consumer to broker
> > 10.189.179.118:9092 (id: 2 rack: null)
> > [2018-08-09 11:40:51] :: DEBUG ::
> NetworkClient$DefaultMetadataUpdater:640
> > - Sending metadata request {topics=[bulk-email]} to node 2
> > [2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster
> metadata
> > version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> > 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
> > null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> > [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> > [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> > leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> > bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> > replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> > partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> > Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> > [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> > leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> > bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> > [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> > replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> > partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> > coordinator response ClientResponse(receivedTimeMs=1533795051063,
> > disconnected=false, request=ClientRequest(expectResponse=true,
> > callback=org.apache.kafka.clients.consumer.internals.
> > ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
> > request=RequestSend(header={api_key=10,api_version=0,
> > correlation_id=349316,client_id=consumer-19},
> > body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
> > sendTimeMs=1533795051061),
> > responseBody={error_code=0,coordinator={node_id=0,host=
> > 10.189.179.116,port=9092}})
> > [2018-08-09 11:40:51] :: INFO ::
> > AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> > coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> > bulk-emai -consumer.
> > [2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> > group bulk-email-consumer
> > [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
> > JoinGroup
> > ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> > 300000,member_id=,protocol_type=consumer,group_protocols=
> > [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> > rack:
> > null)
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:40:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:00] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:17] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:17] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:31] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:31] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:33] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:36] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:39] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:42] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:46] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:46] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:48] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:51] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:54] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:41:57] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:01] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:03] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:06] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:09] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:12] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:15] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:18] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:21] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:24] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:27] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:31] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:34] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:37] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:40] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:43] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:46] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:49] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:52] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:55] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:42:58] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:01] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:04] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:07] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:10] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:13] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:16] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:19] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:22] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:25] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> > failed for group bulk-email-consumer since it is rebalancing.
> > [2018-08-09 11:43:26] :: DEBUG ::
> > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > join
> > group response for group bulk-email-consumer:
> > {error_code=0,generation_id=1225,group_protocol=range,
> > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
> > [2018-08-09 11:43:26] :: DEBUG ::
> > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > join
> > group response for group bulk-email-consumer:
> > {error_code=0,generation_id=1225,group_protocol=range,
> > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
> > [2018-08-09 11:43:26] :: DEBUG ::
> > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > join
> > group response for group bulk-email-consumer:
> > {error_code=0,generation_id=1225,group_protocol=range,
> > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
> > [2018-08-09 11:43:26] :: DEBUG ::
> > AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> > join
> > group response for group bulk-email-consumer:
> > {error_code=0,generation_id=1225,group_protocol=range,
> > leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
> > members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> > 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22
> > cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22
> > cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22
> > cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
> > lim=22 cap=22]}]}
> > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > follower SyncGroup for group bulk-email-consumer to coordinator
> > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
> > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > follower SyncGroup for group bulk-email-consumer to coordinator
> > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
> > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> > follower SyncGroup for group bulk-email-consumer to coordinator
> > 10.189.179.116:9092 (id: 2147483647 rack: null):
> > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 - Performing
> > assignment for group bulk-email-consumer using strategy range with
> > subscriptions
> > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > Subscription(topics=[bulk-email]),
> > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > Subscription(topics=[bulk-email]),
> > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > Subscription(topics=[bulk-email]),
> > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > Subscription(topics=[bulk-email])}
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
> > assignment for group bulk-email-consumer:
> > {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> > Assignment(partitions=[bulk-email-3,
> > bulk-email-4, bulk-email-5]),
> > consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> > Assignment(partitions=[bulk-email-0,
> > bulk-email-1, bulk-email-2]),
> > consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> > Assignment(partitions=[bulk-email-6,
> > bulk-email-7]),
> > consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> > Assignment(partitions=[bulk-email-8,
> > bulk-email-9])}
> > [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending
> leader
> > SyncGroup for group bulk-email-consumer to coordinator
> 10.189.179.116:9092
> > (id: 2147483647 rack: null):
> > {group_id=bulk-email-consumer,generation_id=1225,member_id=
> > consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
> > assignment=[{member_id=consumer-23-bdc073b6-1f90-
> > 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
> > lim=38
> > cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> > 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > lim=38
> > cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> > 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
> > lim=34
> > cap=34]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> > 1c4dd363ba09,member_assignment=java.nio.HeapByteBuffer[pos=0
> > lim=34 cap=34]}]}
> > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> Successfully
> > joined group bulk-email-consumer with generation 1225
> > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> Successfully
> > joined group bulk-email-consumer with generation 1225
> > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> Successfully
> > joined group bulk-email-consumer with generation 1225
> > [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 -
> Successfully
> > joined group bulk-email-consumer with generation 1225
> > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> > assigned partitions [bulk-email-7, bulk-email-6] for group
> > bulk-email-consumer
> > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> > assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1] for group
> > bulk-email-consumer
> > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> > assigned partitions [bulk-email-4, bulk-email-3, bulk-email-5] for group
> > bulk-email-consumer
> > [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> > assigned partitions [bulk-email-8, bulk-email-9] for group
> > bulk-email-consumer
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > bulk-email-consumer fetching committed offsets for partitions:
> > [bulk-email-7, bulk-email-6]
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > bulk-email-consumer fetching committed offsets for partitions:
> > [bulk-email-0, bulk-email-2, bulk-email-1]
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > bulk-email-consumer fetching committed offsets for partitions:
> > [bulk-email-4, bulk-email-3, bulk-email-5]
> > [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> > bulk-email-consumer fetching committed offsets for partitions:
> > [bulk-email-8, bulk-email-9]
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-7 to the committed offset 25435998
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-0 to the committed offset 25390842
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-6 to the committed offset 25463566
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-4 to the committed offset 25409183
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-8 to the committed offset 25465113
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-2 to the committed offset 25368327
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched
> records
> > for partition bulk-email-8 since it is no longer fetchable
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-3 to the committed offset 25422008
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-9 to the committed offset 25444488
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-1 to the committed offset 25369263
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> > partition bulk-email-5 to the committed offset 25456518
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :470 - Ignoring fetched
> records
> > for bulk-email-0 at offset 25390442 since the current position is
> 25390842
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched
> records
> > for partition bulk-email-6 since it is no longer fetchable
> > [2018-08-09 11:43:26] :: DEBUG :: Fetcher :671 - Discarding stale fetch
> > response for partition bulk-email-1 since its offset 25369219 does not
> > match the expected offset 25369263
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> > [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> > metadata=''}, bulk-email-9=OffsetAndMetadata{offset=25444488,
> > metadata=''}}
> > for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25436018 for partition bulk-email-7
> > [2018-08-09 11:43:29] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> > [2018-08-09 11:43:29] :: DEBUG ::
> > AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> > heartbeat response for group bulk-email-consumer
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25409183 for partition bulk-email-4
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25422008 for partition bulk-email-3
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25456530 for partition bulk-email-5
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25390842 for partition bulk-email-0
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25368339 for partition bulk-email-2
> > [2018-08-09 11:43:29] :: DEBUG ::
> > ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> > bulk-email-consumer committed offset 25369263 for partition bulk-email-1
> > [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> > autocommit of offsets {bulk-email-4=OffsetAndMetadata{offset=25409183,
> > metadata=''}, bulk-email-3=OffsetAndMetadata{offset=25422008,
> > metadata=''},
> > bulk-email-5=OffsetAndMetadata{offset=25456530, metadata=''}} for group
> > bulk-email-consumer
> >
> > On Thu, Jul 12, 2018 at 9:27 PM Steve Tian <st...@gmail.com>
> > wrote:
> >
> > > It's a very good and important doc so I think you should read it all.
> > You
> > > should get some idea from sections like *Detecting Consumer Failures*
> and
> > > *Multi-threaded Processing* for your case.
> > >
> > > On Thu, Jul 12, 2018, 3:17 PM Shantanu Deshmukh <shantanu88d@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Steve,
> > > >
> > > > Could you please shed more light on this? What section should I
> > revisit?
> > > I
> > > > am using high-level consumer. So I am simply calling consumer.close()
> > > when
> > > > I am shutting down the process. Is there any other method to be
> called
> > > > before calling close()?
> > > >
> > > > On Mon, Jul 9, 2018 at 5:58 PM Steve Tian <st...@gmail.com>
> > > wrote:
> > > >
> > > > > Please re-read the javadoc of KafkaConsumer, make sure you know how
> > to
> > > > > wakeup/close consumer properly while shutting down your
> application.
> > > Try
> > > > > to understand the motivation of KIP-62 and adjust related timeout.
> > > > >
> > > > > On Mon, Jul 9, 2018, 8:05 PM harish lohar <hk...@gmail.com>
> wrote:
> > > > >
> > > > > > Try reducing below timer
> > > > > > metadata.max.age.ms = 300000
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 6, 2018 at 5:55 AM Shantanu Deshmukh <
> > > > shantanu88d@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a
> java
> > > app
> > > > > > which
> > > > > > > spawns many consumer threads consuming from different topics.
> For
> > > > every
> > > > > > > topic we have specified different consumer-group. A lot of
> times
> > I
> > > > see
> > > > > > that
> > > > > > > whenever this application is restarted a CG on one or two
> topics
> > > > takes
> > > > > > more
> > > > > > > than 5 minutes to receive partition assignment. Till that time
> > > > > consumers
> > > > > > > for that topic don't consumer anything. If I go to Kafka broker
> > and
> > > > run
> > > > > > > consumer-groups.sh and describe that particular CG I see that
> it
> > is
> > > > > > > rebalancing. There is time critical data stored in that topic
> and
> > > we
> > > > > > cannot
> > > > > > > tolerate such long delays. What can be the reason for such long
> > > > > > rebalances.
> > > > > > >
> > > > > > > Here's our consumer config
> > > > > > >
> > > > > > >
> > > > > > > auto.commit.interval.ms = 3000
> > > > > > > auto.offset.reset = latest
> > > > > > > bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092, x.x.x.x:9092]
> > > > > > > check.crcs = true
> > > > > > > client.id =
> > > > > > > connections.max.idle.ms = 540000
> > > > > > > enable.auto.commit = true
> > > > > > > exclude.internal.topics = true
> > > > > > > fetch.max.bytes = 52428800
> > > > > > > fetch.max.wait.ms = 500
> > > > > > > fetch.min.bytes = 1
> > > > > > > group.id = otp-notifications-consumer
> > > > > > > heartbeat.interval.ms = 3000
> > > > > > > interceptor.classes = null
> > > > > > > key.deserializer = class
> > > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > > max.partition.fetch.bytes = 1048576
> > > > > > > max.poll.interval.ms = 300000
> > > > > > > max.poll.records = 50
> > > > > > > metadata.max.age.ms = 300000
> > > > > > > metric.reporters = []
> > > > > > > metrics.num.samples = 2
> > > > > > > metrics.sample.window.ms = 30000
> > > > > > > partition.assignment.strategy = [class
> > > > > > > org.apache.kafka.clients.consumer.RangeAssignor]
> > > > > > > receive.buffer.bytes = 65536
> > > > > > > reconnect.backoff.ms = 50
> > > > > > > request.timeout.ms = 305000
> > > > > > > retry.backoff.ms = 100
> > > > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit
> > > > > > > sasl.kerberos.min.time.before.relogin = 60000
> > > > > > > sasl.kerberos.service.name = null
> > > > > > > sasl.kerberos.ticket.renew.jitter = 0.05
> > > > > > > sasl.kerberos.ticket.renew.window.factor = 0.8
> > > > > > > sasl.mechanism = GSSAPI
> > > > > > > security.protocol = SSL
> > > > > > > send.buffer.bytes = 131072
> > > > > > > session.timeout.ms = 300000
> > > > > > > ssl.cipher.suites = null
> > > > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> > > > > > > ssl.endpoint.identification.algorithm = null
> > > > > > > ssl.key.password = null
> > > > > > > ssl.keymanager.algorithm = SunX509
> > > > > > > ssl.keystore.location = null
> > > > > > > ssl.keystore.password = null
> > > > > > > ssl.keystore.type = JKS
> > > > > > > ssl.protocol = TLS
> > > > > > > ssl.provider = null
> > > > > > > ssl.secure.random.implementation = null
> > > > > > > ssl.trustmanager.algorithm = PKIX
> > > > > > > ssl.truststore.location = /x/x/client.truststore.jks
> > > > > > > ssl.truststore.password = [hidden]
> > > > > > > ssl.truststore.type = JKS
> > > > > > > value.deserializer = class
> > > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > >
> > > > > > > Please help.
> > > > > > >
> > > > > > > *Thanks & Regards,*
> > > > > > > *Shantanu Deshmukh*
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Very long consumer rebalances
Posted by "M. Manna" <ma...@gmail.com>.
In the simplest way, how have you implemented your consumer?
1) Does your consumers join a designated group, process messages, and then
closes all connection? Or does it stay open perpetually until server
shutdown?
2) Have you configured the session timeouts for client and zookeeper
accordingly?
Regards,
On 9 August 2018 at 08:00, Shantanu Deshmukh <sh...@gmail.com> wrote:
> I am facing too many problems these days. Now one of our consumer groups
> is rebalancing every now and then. And rebalance takes very low, more than
> 5-10 minutes. Even after re-balancing I see that only half of the consumers
> are active/receive assignment. Its all going haywire.
>
> I am seeing these logs in kafka consumer logs. Can anyone help me
> understand what is going on here? It is a very long piece of log, but
> someone please help me. I am desperately looking for any solution since
> more than 2 months now. But to no avail.
>
> [2018-08-09 11:39:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:53] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:53] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:53] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:39:53] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:56] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:39:56] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:39:56] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:56] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:39:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
> group bulk-email-consumer failed: The request timed out.
> [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
> bulk-email-consumer
> [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
> commit failed for group bulk-email-consumer: Offset commit failed with a
> retriable exception. You should retry committing offsets.
> [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> coordinator request for group bulk-email-consumer to broker
> 10.189.179.117:9092 (id: 1 rack: null)
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:664 - Offset commit for
> group bulk-email-consumer failed: The request timed out.
> [2018-08-09 11:40:04] :: INFO :: AbstractCoordinator:600 - Marking the
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
> bulk-email-consumer
> [2018-08-09 11:40:04] :: WARN :: ConsumerCoordinator$4:535 - Auto offset
> commit failed for group bulk-email-consumer: Offset commit failed with a
> retriable exception. You should retry committing offsets.
> [2018-08-09 11:40:04] :: DEBUG :: AbstractCoordinator:531 - Sending
> coordinator request for group bulk-email-consumer to broker
> 10.189.179.118:9092 (id: 2 rack: null)
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> coordinator response ClientResponse(receivedTimeMs=1533795004158,
> disconnected=false, request=ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@3a3e5740,
> request=RequestSend(header={api_key=10,api_version=0,
> correlation_id=389784,client_id=consumer-26},
> body={group_id=bulk-email-consumer}), createdTimeMs=1533795004157,
> sendTimeMs=1533795004157),
> responseBody={error_code=0,coordinator={node_id=0,host=
> 10.189.179.116,port=9092}})
> [2018-08-09 11:40:04] :: INFO ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> bulk-emai -consumer.
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> coordinator response ClientResponse(receivedTimeMs=1533795004718,
> disconnected=false, request=ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@66889bbd,
> request=RequestSend(header={api_key=10,api_version=0,
> correlation_id=363859,client_id=consumer-25},
> body={group_id=bulk-email-consumer}), createdTimeMs=1533795004145,
> sendTimeMs=1533795004145),
> responseBody={error_code=0,coordinator={node_id=0,host=
> 10.189.179.116,port=9092}})
> [2018-08-09 11:40:04] :: INFO ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> bulk-emai -consumer.
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:04] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:04] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:05] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:05] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:07] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:07] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:07] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:07] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:07] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:08] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:08] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:10] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:10] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:10] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:10] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:10] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:11] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:11] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:13] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:13] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:13] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:14] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:14] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:14] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:14] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:16] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:16] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:16] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:16] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:16] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:17] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:17] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:19] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:19] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:19] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:19] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:19] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:40:20] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:20] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-6=OffsetAndMetadata{offset=25463566,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:21] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:40:21] :: INFO :: ConsumerCoordinator:333 - Revoking
> previously assigned partitions [bulk-email-6] for group bulk-email-consumer
> [2018-08-09 11:40:21] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:21] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> 869377f9ead1,protocol_type=consumer,group_protocols=[{
> protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:22] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:22] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:22] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}} for group bulk-email-consumer
> [2018-08-09 11:40:22] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:22] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:22] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:40:22] :: INFO :: ConsumerCoordinator:333 - Revoking
> previously assigned partitions [bulk-email-9] for group bulk-email-consumer
> [2018-08-09 11:40:22] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:22] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=consumer-26-f5a1293d-7d14-4519-8378-
> 1c4dd363ba09,protocol_type=consumer,group_protocols=[{
> protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:22] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:40:23] :: INFO :: ConsumerCoordinator:333 - Revoking
> previously assigned partitions [bulk-email-8] for group bulk-email-consumer
> [2018-08-09 11:40:23] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:23] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=consumer-25-fa10026f-031e-49aa-9df1-
> 9d083b256ffa,protocol_type=consumer,group_protocols=[{
> protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:26] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 1
> [2018-08-09 11:40:26] :: DEBUG :: Metadata :241 - Updated cluster metadata
> version 642 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> 10.189.179.118:9092 (id: 2 rack: null), 10.189.179.116:9092 (id: 0 rack:
> null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:28] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 1
> [2018-08-09 11:40:28] :: DEBUG :: Metadata :241 - Updated cluster metadata
> version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> 10.189.179.116:9092 (id: 0 rack: null), 10.189.179.118:9092 (id: 2 rack:
> null), 10.189.179.117:9092 (id: 1 rack: null)], partitions =
> [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> [2018-08-09 11:40:30] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:30] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:30] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:30] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:31] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:34] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 1
> [2018-08-09 11:40:34] :: DEBUG :: Metadata :241 - Updated cluster metadata
> version 646 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
> null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:45] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:50] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-0=OffsetAndMetadata{offset=25390438,
> metadata=''}, bulk-email-2=OffsetAndMetadata{offset=25368327,
> metadata=''},
> bulk-email-1=OffsetAndMetadata{offset=25369219, metadata=''}} for group
> bulk-email-consumer
> [2018-08-09 11:40:50] :: WARN :: ConsumerCoordinator:554 - Auto offset
> commit failed for group bulk-email-consumer: Commit cannot be completed
> since the group has already rebalanced and assigned the partitions to
> another member. This means that the time between subsequent calls to poll()
> was longer than the configured max.poll.interval.ms, which typically
> implies that the poll loop is spending too much time message processing.
> You can address this either by increasing the session timeout or by
> reducing the maximum size of batches returned in poll() with
> max.poll.records.
> [2018-08-09 11:40:50] :: INFO :: ConsumerCoordinator:333 - Revoking
> previously assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1]
> for group bulk-email-consumer
> [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=,protocol_type=consumer,group_protocols=
> [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:50] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 0
> [2018-08-09 11:40:50] :: DEBUG ::
> ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> LEAVE_GROUP request ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@60f41e0b,
> request=RequestSend(header={api_key=13,api_version=0,
> correlation_id=349313,client_id=consumer-19},
> body={group_id=bulk-email-consumer,member_id=consumer-
> 19-34903180-9cb3-496d-a913-b782476d9a52}),
> createdTimeMs=1533794419195, sendTimeMs=1533794419195) with correlation id
> 349313 due to node 2147483647 being disconnected
> [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
> bulk-email-consumer
> [2018-08-09 11:40:50] :: DEBUG ::
> ConsumerNetworkClient$RequestFutureCompletionHandler:475 - Cancelled
> JOIN_GROUP request ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@3bd6f5c9,
> request=RequestSend(header={api_key=11,api_version=1,
> correlation_id=349314,client_id=consumer-19},
> body={group_id=bulk-email-consumer,session_timeout=
> 300000,rebalance_timeout=300000,member_id=,protocol_
> type=consumer,group_protocols=[{protocol_name=range,
> protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}), createdTimeMs=1533795050857, sendTimeMs=0) with
> correlation id 349314 due to node 2147483647 being disconnected
> [2018-08-09 11:40:50] :: INFO :: AbstractCoordinator:600 - Marking the
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) dead for group
> bulk-email-consumer
> [2018-08-09 11:40:50] :: DEBUG :: AbstractCoordinator:531 - Sending
> coordinator request for group bulk-email-consumer to broker
> 10.189.179.118:9092 (id: 2 rack: null)
> [2018-08-09 11:40:51] :: DEBUG :: NetworkClient$DefaultMetadataUpdater:640
> - Sending metadata request {topics=[bulk-email]} to node 2
> [2018-08-09 11:40:51] :: DEBUG :: Metadata :241 - Updated cluster metadata
> version 641 to Cluster(id = 09FjKmqPQvat0c0dyPptIw, nodes = [
> 10.189.179.117:9092 (id: 1 rack: null), 10.189.179.116:9092 (id: 0 rack:
> null), 10.189.179.118:9092 (id: 2 rack: null)], partitions =
> [Partition(topic = bulk-email, partition = 8, leader = 2, replicas =
> [0,1,2,], isr = [0,2,1,]), Partition(topic = bulk-email, partition = 7,
> leader = 1, replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic =
> bulk-email, partition = 9, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 4, leader = 1,
> replicas = [0,1,2,], isr = [1,0,2,]), Partition(topic = bulk-email,
> partition = 3, leader = 0, replicas = [0,1,2,], isr = [0,1,2,]),
> Partition(topic = bulk-email, partition = 6, leader = 0, replicas =
> [0,1,2,], isr = [0,1,2,]), Partition(topic = bulk-email, partition = 5,
> leader = 2, replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic =
> bulk-email, partition = 0, leader = 0, replicas = [0,1,2,], isr =
> [0,1,2,]), Partition(topic = bulk-email, partition = 2, leader = 2,
> replicas = [0,1,2,], isr = [2,0,1,]), Partition(topic = bulk-email,
> partition = 1, leader = 1, replicas = [0,1,2,], isr = [1,0,2,])])
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:541 - Received group
> coordinator response ClientResponse(receivedTimeMs=1533795051063,
> disconnected=false, request=ClientRequest(expectResponse=true,
> callback=org.apache.kafka.clients.consumer.internals.
> ConsumerNetworkClient$RequestFutureCompletionHandler@21a34a4e,
> request=RequestSend(header={api_key=10,api_version=0,
> correlation_id=349316,client_id=consumer-19},
> body={group_id=bulk-email-consumer}), createdTimeMs=1533795050961,
> sendTimeMs=1533795051061),
> responseBody={error_code=0,coordinator={node_id=0,host=
> 10.189.179.116,port=9092}})
> [2018-08-09 11:40:51] :: INFO ::
> AbstractCoordinator$GroupCoordinatorResponseHandler:555 - Discovered
> coordinator 10.189.179.116:9092 (id: 2147483647 rack: null) for group
> bulk-emai -consumer.
> [2018-08-09 11:40:51] :: INFO :: AbstractCoordinator:381 - (Re-)joining
> group bulk-email-consumer
> [2018-08-09 11:40:51] :: DEBUG :: AbstractCoordinator:390 - Sending
> JoinGroup
> ({group_id=bulk-email-consumer,session_timeout=300000,rebalance_timeout=
> 300000,member_id=,protocol_type=consumer,group_protocols=
> [{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}) to coordinator 10.189.179.116:9092 (id: 2147483647
> rack:
> null)
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:40:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:00] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:17] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:17] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:31] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:31] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:33] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:36] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:39] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:42] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:46] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:46] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:48] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:51] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:54] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:41:57] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:01] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:03] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:06] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:09] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:12] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:15] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:18] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:21] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:24] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:27] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:31] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:34] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:37] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:40] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:43] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:46] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:49] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:52] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:55] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:42:58] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:01] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:04] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:07] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:10] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:13] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:16] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:19] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:22] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:25] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:703 - Attempt to heart beat
> failed for group bulk-email-consumer since it is rebalancing.
> [2018-08-09 11:43:26] :: DEBUG ::
> AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> join
> group response for group bulk-email-consumer:
> {error_code=0,generation_id=1225,group_protocol=range,
> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,members=[]}
> [2018-08-09 11:43:26] :: DEBUG ::
> AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> join
> group response for group bulk-email-consumer:
> {error_code=0,generation_id=1225,group_protocol=range,
> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,members=[]}
> [2018-08-09 11:43:26] :: DEBUG ::
> AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> join
> group response for group bulk-email-consumer:
> {error_code=0,generation_id=1225,group_protocol=range,
> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,members=[]}
> [2018-08-09 11:43:26] :: DEBUG ::
> AbstractCoordinator$JoinGroupResponseHandler:406 - Received successful
> join
> group response for group bulk-email-consumer:
> {error_code=0,generation_id=1225,group_protocol=range,
> leader_id=consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,member_id=
> consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,
> members=[{member_id=consumer-23-bdc073b6-1f90-4ac5-a801-
> 869377f9ead1,member_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22
> cap=250]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> 6f0621f10afa,member_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22
> cap=174]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> 9d083b256ffa,member_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22
> cap=98]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> 1c4dd363ba09,member_metadata=java.nio.HeapByteBuffer[pos=0
> lim=22 cap=22]}]}
> [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> follower SyncGroup for group bulk-email-consumer to coordinator
> 10.189.179.116:9092 (id: 2147483647 rack: null):
> {group_id=bulk-email-consumer,generation_id=1225,member_id=
> consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09,group_assignment=[]}
> [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> follower SyncGroup for group bulk-email-consumer to coordinator
> 10.189.179.116:9092 (id: 2147483647 rack: null):
> {group_id=bulk-email-consumer,generation_id=1225,member_id=
> consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa,group_assignment=[]}
> [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:460 - Sending
> follower SyncGroup for group bulk-email-consumer to coordinator
> 10.189.179.116:9092 (id: 2147483647 rack: null):
> {group_id=bulk-email-consumer,generation_id=1225,member_id=
> consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa,group_assignment=[]}
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:310 - Performing
> assignment for group bulk-email-consumer using strategy range with
> subscriptions
> {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> Subscription(topics=[bulk-email]),
> consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> Subscription(topics=[bulk-email]),
> consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> Subscription(topics=[bulk-email]),
> consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> Subscription(topics=[bulk-email])}
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:315 - Finished
> assignment for group bulk-email-consumer:
> {consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1=
> Assignment(partitions=[bulk-email-3,
> bulk-email-4, bulk-email-5]),
> consumer-19-59fa9871-2dcc-4713-9c4c-6f0621f10afa=
> Assignment(partitions=[bulk-email-0,
> bulk-email-1, bulk-email-2]),
> consumer-25-fa10026f-031e-49aa-9df1-9d083b256ffa=
> Assignment(partitions=[bulk-email-6,
> bulk-email-7]),
> consumer-26-f5a1293d-7d14-4519-8378-1c4dd363ba09=
> Assignment(partitions=[bulk-email-8,
> bulk-email-9])}
> [2018-08-09 11:43:26] :: DEBUG :: AbstractCoordinator:471 - Sending leader
> SyncGroup for group bulk-email-consumer to coordinator 10.189.179.116:9092
> (id: 2147483647 rack: null):
> {group_id=bulk-email-consumer,generation_id=1225,member_id=
> consumer-23-bdc073b6-1f90-4ac5-a801-869377f9ead1,group_
> assignment=[{member_id=consumer-23-bdc073b6-1f90-
> 4ac5-a801-869377f9ead1,member_assignment=java.nio.HeapByteBuffer[pos=0
> lim=38
> cap=38]},{member_id=consumer-19-59fa9871-2dcc-4713-9c4c-
> 6f0621f10afa,member_assignment=java.nio.HeapByteBuffer[pos=0
> lim=38
> cap=38]},{member_id=consumer-25-fa10026f-031e-49aa-9df1-
> 9d083b256ffa,member_assignment=java.nio.HeapByteBuffer[pos=0
> lim=34
> cap=34]},{member_id=consumer-26-f5a1293d-7d14-4519-8378-
> 1c4dd363ba09,member_assignment=java.nio.HeapByteBuffer[pos=0
> lim=34 cap=34]}]}
> [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
> joined group bulk-email-consumer with generation 1225
> [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
> joined group bulk-email-consumer with generation 1225
> [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
> joined group bulk-email-consumer with generation 1225
> [2018-08-09 11:43:26] :: INFO :: AbstractCoordinator$1:349 - Successfully
> joined group bulk-email-consumer with generation 1225
> [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> assigned partitions [bulk-email-7, bulk-email-6] for group
> bulk-email-consumer
> [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> assigned partitions [bulk-email-0, bulk-email-2, bulk-email-1] for group
> bulk-email-consumer
> [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> assigned partitions [bulk-email-4, bulk-email-3, bulk-email-5] for group
> bulk-email-consumer
> [2018-08-09 11:43:26] :: INFO :: ConsumerCoordinator:225 - Setting newly
> assigned partitions [bulk-email-8, bulk-email-9] for group
> bulk-email-consumer
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> bulk-email-consumer fetching committed offsets for partitions:
> [bulk-email-7, bulk-email-6]
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> bulk-email-consumer fetching committed offsets for partitions:
> [bulk-email-0, bulk-email-2, bulk-email-1]
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> bulk-email-consumer fetching committed offsets for partitions:
> [bulk-email-4, bulk-email-3, bulk-email-5]
> [2018-08-09 11:43:26] :: DEBUG :: ConsumerCoordinator:708 - Group
> bulk-email-consumer fetching committed offsets for partitions:
> [bulk-email-8, bulk-email-9]
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-7 to the committed offset 25435998
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-0 to the committed offset 25390842
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-6 to the committed offset 25463566
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-4 to the committed offset 25409183
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-8 to the committed offset 25465113
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-2 to the committed offset 25368327
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched records
> for partition bulk-email-8 since it is no longer fetchable
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-3 to the committed offset 25422008
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-9 to the committed offset 25444488
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-1 to the committed offset 25369263
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :221 - Resetting offset for
> partition bulk-email-5 to the committed offset 25456518
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :470 - Ignoring fetched records
> for bulk-email-0 at offset 25390442 since the current position is 25390842
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :665 - Ignoring fetched records
> for partition bulk-email-6 since it is no longer fetchable
> [2018-08-09 11:43:26] :: DEBUG :: Fetcher :671 - Discarding stale fetch
> response for partition bulk-email-1 since its offset 25369219 does not
> match the expected offset 25369263
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25465113 for partition bulk-email-8
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25444488 for partition bulk-email-9
> [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-8=OffsetAndMetadata{offset=25465113,
> metadata=''}, bulk-email-9=OffsetAndMetadata{offset=25444488,
> metadata=''}}
> for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25436018 for partition bulk-email-7
> [2018-08-09 11:43:29] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25463566 for partition bulk-email-6
> [2018-08-09 11:43:29] :: DEBUG ::
> AbstractCoordinator$HeartbeatResponseHandler:694 - Received successful
> heartbeat response for group bulk-email-consumer
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25409183 for partition bulk-email-4
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25422008 for partition bulk-email-3
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25456530 for partition bulk-email-5
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25390842 for partition bulk-email-0
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25368339 for partition bulk-email-2
> [2018-08-09 11:43:29] :: DEBUG ::
> ConsumerCoordinator$OffsetCommitResponseHandler:640 - Group
> bulk-email-consumer committed offset 25369263 for partition bulk-email-1
> [2018-08-09 11:43:29] :: DEBUG :: ConsumerCoordinator$4:539 - Completed
> autocommit of offsets {bulk-email-4=OffsetAndMetadata{offset=25409183,
> metadata=''}, bulk-email-3=OffsetAndMetadata{offset=25422008,
> metadata=''},
> bulk-email-5=OffsetAndMetadata{offset=25456530, metadata=''}} for group
> bulk-email-consumer
>
> On Thu, Jul 12, 2018 at 9:27 PM Steve Tian <st...@gmail.com>
> wrote:
>
> > It's a very good and important doc so I think you should read it all.
> You
> > should get some idea from sections like *Detecting Consumer Failures* and
> > *Multi-threaded Processing* for your case.
> >
> > On Thu, Jul 12, 2018, 3:17 PM Shantanu Deshmukh <sh...@gmail.com>
> > wrote:
> >
> > > Hi Steve,
> > >
> > > Could you please shed more light on this? What section should I
> revisit?
> > I
> > > am using high-level consumer. So I am simply calling consumer.close()
> > when
> > > I am shutting down the process. Is there any other method to be called
> > > before calling close()?
> > >
> > > On Mon, Jul 9, 2018 at 5:58 PM Steve Tian <st...@gmail.com>
> > wrote:
> > >
> > > > Please re-read the javadoc of KafkaConsumer, make sure you know how
> to
> > > > wakeup/close consumer properly while shutting down your application.
> > Try
> > > > to understand the motivation of KIP-62 and adjust related timeout.
> > > >
> > > > On Mon, Jul 9, 2018, 8:05 PM harish lohar <hk...@gmail.com> wrote:
> > > >
> > > > > Try reducing below timer
> > > > > metadata.max.age.ms = 300000
> > > > >
> > > > >
> > > > > On Fri, Jul 6, 2018 at 5:55 AM Shantanu Deshmukh <
> > > shantanu88d@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a java
> > app
> > > > > which
> > > > > > spawns many consumer threads consuming from different topics. For
> > > every
> > > > > > topic we have specified different consumer-group. A lot of times
> I
> > > see
> > > > > that
> > > > > > whenever this application is restarted a CG on one or two topics
> > > takes
> > > > > more
> > > > > > than 5 minutes to receive partition assignment. Till that time
> > > > consumers
> > > > > > for that topic don't consumer anything. If I go to Kafka broker
> and
> > > run
> > > > > > consumer-groups.sh and describe that particular CG I see that it
> is
> > > > > > rebalancing. There is time critical data stored in that topic and
> > we
> > > > > cannot
> > > > > > tolerate such long delays. What can be the reason for such long
> > > > > rebalances.
> > > > > >
> > > > > > Here's our consumer config
> > > > > >
> > > > > >
> > > > > > auto.commit.interval.ms = 3000
> > > > > > auto.offset.reset = latest
> > > > > > bootstrap.servers = [x.x.x.x:9092, x.x.x.x:9092, x.x.x.x:9092]
> > > > > > check.crcs = true
> > > > > > client.id =
> > > > > > connections.max.idle.ms = 540000
> > > > > > enable.auto.commit = true
> > > > > > exclude.internal.topics = true
> > > > > > fetch.max.bytes = 52428800
> > > > > > fetch.max.wait.ms = 500
> > > > > > fetch.min.bytes = 1
> > > > > > group.id = otp-notifications-consumer
> > > > > > heartbeat.interval.ms = 3000
> > > > > > interceptor.classes = null
> > > > > > key.deserializer = class
> > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > > max.partition.fetch.bytes = 1048576
> > > > > > max.poll.interval.ms = 300000
> > > > > > max.poll.records = 50
> > > > > > metadata.max.age.ms = 300000
> > > > > > metric.reporters = []
> > > > > > metrics.num.samples = 2
> > > > > > metrics.sample.window.ms = 30000
> > > > > > partition.assignment.strategy = [class
> > > > > > org.apache.kafka.clients.consumer.RangeAssignor]
> > > > > > receive.buffer.bytes = 65536
> > > > > > reconnect.backoff.ms = 50
> > > > > > request.timeout.ms = 305000
> > > > > > retry.backoff.ms = 100
> > > > > > sasl.kerberos.kinit.cmd = /usr/bin/kinit
> > > > > > sasl.kerberos.min.time.before.relogin = 60000
> > > > > > sasl.kerberos.service.name = null
> > > > > > sasl.kerberos.ticket.renew.jitter = 0.05
> > > > > > sasl.kerberos.ticket.renew.window.factor = 0.8
> > > > > > sasl.mechanism = GSSAPI
> > > > > > security.protocol = SSL
> > > > > > send.buffer.bytes = 131072
> > > > > > session.timeout.ms = 300000
> > > > > > ssl.cipher.suites = null
> > > > > > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> > > > > > ssl.endpoint.identification.algorithm = null
> > > > > > ssl.key.password = null
> > > > > > ssl.keymanager.algorithm = SunX509
> > > > > > ssl.keystore.location = null
> > > > > > ssl.keystore.password = null
> > > > > > ssl.keystore.type = JKS
> > > > > > ssl.protocol = TLS
> > > > > > ssl.provider = null
> > > > > > ssl.secure.random.implementation = null
> > > > > > ssl.trustmanager.algorithm = PKIX
> > > > > > ssl.truststore.location = /x/x/client.truststore.jks
> > > > > > ssl.truststore.password = [hidden]
> > > > > > ssl.truststore.type = JKS
> > > > > > value.deserializer = class
> > > > > > org.apache.kafka.common.serialization.StringDeserializer
> > > > > >
> > > > > > Please help.
> > > > > >
> > > > > > *Thanks & Regards,*
> > > > > > *Shantanu Deshmukh*
> > > > > >
> > > > >
> > > >
> > >
> >
>