You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ann Pricks <pr...@dt.ae> on 2020/07/03 12:09:43 UTC
Consumer Groups Describe is not working
Hi Team,
Today, In our production cluster, we faced an issue with Kafka (Old offsets was getting pulled from spark streaming application) and couldn't debug the issue using kafka_consumer_group.sh CLI.
Whenever we execute the below command to list the consumer groups, it is working fine. However, whenever we try to describe the consumer group to get to know the offset details, it didn't work (Nothing is getting displayed. Just blank).
Command to list the consumer group (Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--list \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Command to list the consumer group (Not Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--describe \
--group spark-kafka-source-f8e218d5-16d2-4e63-a25c-2f96fabb2809-605351645-driver-0 \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Jass Config File:
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="admin" \
password="123@admin";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
exclude.internal.topics=false
Kindly help us to monitor our Kafka cluster in case of any issues.
Details:
Kafka Version: 2.0.0
Security:
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
security.inter.broker.protocol=SASL_PLAINTEXT
Please let us know in case of any other details required from our end.
Regards,
AnnPricksEdmund
Ann Pricks
DBA Admin
BIG DATA
[https://www.dt.ae/signature/divider.jpg]
[Map] DT Building 8th Street, Al Khabaisi, 4286, Dubai, AE
[Phone] Ext: 152
[Phone] +971 4 222 4222
[Email] pricks@dt.ae
[Web] www.dt.ae
[https://www.dt.ae/signature/free-logo%20copy.png]
[https://www.facebook.com/DubaiTechnologies/]<https://www.facebook.com/DubaiTechnologies/> [Twitter] <https://twitter.com/DubaiTech1994> [LinkedIN] <https://www.linkedin.com/in/dubai-technologies-b0884a125> [Instagram] <https://www.instagram.com/dubaitechnologies/> [Instagram] <https://www.youtube.com/channel/UCR5VDdt7lxlf12p6UX8u0sg>
Re: Consumer Groups Describe is not working
Posted by Ann Pricks <pr...@dt.ae>.
Hi Liam,
Thanks for the response.
As we are using Spark Structured Streaming, the commit won't happen at Kafka Side. For checkpoint, we are using HDFS.
As we are expecting, Kafka-consumer-groups.sh CLI should return LOG-END-OFFSET with Partition details. However, it didn't display anything.
So, my question is, why is the CLI not showing any results?
Kindly note that, we are getting LOG-END-OFFSET details in our dev cluster without enabling the security.
Regards,
AnnPricksEdmund
On 09/07/2020, 1:06 AM, "Liam Clarke-Hutchinson" <li...@adscale.co.nz> wrote:
Hi Ann,
It's common practice in many Spark Streaming apps to store offsets external
to Kafka. Especially when checkpointing is enabled.
Are you sure that the app is committing offsets to Kafka?
Kind regards,
Liam Clarke
On Thu, 9 Jul. 2020, 8:00 am Ann Pricks, <pr...@dt.ae> wrote:
> Hi Ricardo,
>
> Thanks for your kind response.
>
> As per your suggestion, I have enabled trace and PFB the content of the
> log file.
>
> Log File Content:
>
> [2020-07-08 18:48:08,963] INFO Registered kafka:type=kafka.Log4jController
> MBean (kafka.utils.Log4jControllerRegistration$)
> [2020-07-08 18:48:09,244] TRACE Registered metric named MetricName
> [name=count, group=kafka-metrics-count, description=total number of
> registered metrics, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,289] INFO Successfully logged in.
> (org.apache.kafka.common.security.authenticator.AbstractLogin)
> [2020-07-08 18:48:09,290] TRACE LoginManager(serviceName=kafka,
> publicCredentials=[admin], refCount=1) acquired
> (org.apache.kafka.common.security.authenticator.LoginManager)
> [2020-07-08 18:48:09,292] DEBUG Updated cluster metadata version 1 to
> Cluster(id = null, nodes = [broker1:2345 (id: -1 rack: null), broker2:2345
> (id: -2 rack: null), broker3:2345 (id: -3 rack: null)], partitions = [],
> controller = null) (org.apache.kafka.clients.Metadata)
> [2020-07-08 18:48:09,303] DEBUG Added sensor with name connections-closed:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,305] TRACE Registered metric named MetricName
> [name=connection-close-total, group=admin-metrics, description=The total
> number of connections closed, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=connection-close-rate, group=admin-metrics, description=The number of
> connections closed per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] DEBUG Added sensor with name
> connections-created: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=connection-creation-total, group=admin-metrics, description=The total
> number of new connections established, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=connection-creation-rate, group=admin-metrics, description=The number
> of new connections established per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] DEBUG Added sensor with name
> successful-authentication: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=successful-authentication-total, group=admin-metrics, description=The
> total number of connections with successful authentication, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=successful-authentication-rate, group=admin-metrics, description=The
> number of connections with successful authentication per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] DEBUG Added sensor with name
> failed-authentication: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=failed-authentication-total, group=admin-metrics, description=The
> total number of connections with failed authentication, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] TRACE Registered metric named MetricName
> [name=failed-authentication-rate, group=admin-metrics, description=The
> number of connections with failed authentication per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] DEBUG Added sensor with name
> bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] TRACE Registered metric named MetricName
> [name=network-io-total, group=admin-metrics, description=The total number
> of network operations (reads or writes) on all connections, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] TRACE Registered metric named MetricName
> [name=network-io-rate, group=admin-metrics, description=The number of
> network operations (reads or writes) on all connections per second,
> tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] DEBUG Added sensor with name bytes-sent:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-metrics, description=The total
> number of outgoing bytes sent to all servers, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,308] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-metrics, description=The number of
> outgoing bytes sent to all servers per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,308] TRACE Registered metric named MetricName
> [name=request-total, group=admin-metrics, description=The total number of
> requests sent, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,308] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-metrics, description=The number of requests
> sent per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,308] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-metrics, description=The average size
> of requests sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-metrics, description=The maximum size
> of any request sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] DEBUG Added sensor with name bytes-received:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-metrics, description=The total
> number of bytes read off all sockets, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-metrics, description=The number of
> bytes read off all sockets per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=response-total, group=admin-metrics, description=The total number of
> responses received, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-metrics, description=The number of
> responses received per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] DEBUG Added sensor with name select-time:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=select-total, group=admin-metrics, description=The total number of
> times the I/O layer checked for new I/O to perform, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=select-rate, group=admin-metrics, description=The number of times the
> I/O layer checked for new I/O to perform per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-wait-time-ns-avg, group=admin-metrics, description=The average
> length of time the I/O thread spent waiting for a socket ready for reads or
> writes in nanoseconds., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-waittime-total, group=admin-metrics, description=The total time
> the I/O thread spent waiting, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-wait-ratio, group=admin-metrics, description=The fraction of time
> the I/O thread spent waiting, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] DEBUG Added sensor with name io-time:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-time-ns-avg, group=admin-metrics, description=The average length
> of time for I/O per select call in nanoseconds., tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=iotime-total, group=admin-metrics, description=The total time the I/O
> thread spent doing I/O, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-ratio, group=admin-metrics, description=The fraction of time the
> I/O thread spent doing I/O, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,312] TRACE Registered metric named MetricName
> [name=connection-count, group=admin-metrics, description=The current number
> of active connections., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,340] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,340] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,340] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker1:2345 (id: -1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,340] DEBUG [NetworkClient clientId=admin-1]
> Initialize connection to node broker1:2345 (id: -1 rack: null) for sending
> metadata request (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,340] DEBUG [NetworkClient clientId=admin-1]
> Initiating connection to node broker1:2345 (id: -1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,351] DEBUG Set SASL client state to
> SEND_APIVERSIONS_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,353] DEBUG Creating SaslClient:
> client=null;service=kafka;serviceHostname=broker1;mechs=[SCRAM-SHA-512]
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,375] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> SEND_CLIENT_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,378] DEBUG Added sensor with name node--1.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] DEBUG Added sensor with name
> node--1.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,384] DEBUG Added sensor with name node--1.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,384] TRACE Registered metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,384] TRACE Registered metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,385] DEBUG [Producer clientId=admin-1] Created socket
> with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
> (org.apache.kafka.common.network.Selector)
> [2020-07-08 18:48:09,452] DEBUG Set SASL client state to
> RECEIVE_APIVERSIONS_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,452] DEBUG [NetworkClient clientId=admin-1] Completed
> connection to node -1. Fetching API versions.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker3:2345 (id: -3 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] DEBUG [NetworkClient clientId=admin-1]
> Initialize connection to node broker3:2345 (id: -3 rack: null) for sending
> metadata request (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] DEBUG [NetworkClient clientId=admin-1]
> Initiating connection to node broker3:2345 (id: -3 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,454] DEBUG Set SASL client state to
> SEND_APIVERSIONS_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,454] DEBUG Creating SaslClient:
> client=null;service=kafka;serviceHostname=broker3;mechs=[SCRAM-SHA-512]
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,455] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> SEND_CLIENT_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,455] DEBUG Added sensor with name node--3.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,455] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,455] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,455] TRACE Registered metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] DEBUG Added sensor with name
> node--3.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,457] DEBUG Added sensor with name node--3.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,457] TRACE Registered metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,457] TRACE Registered metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,457] DEBUG [Producer clientId=admin-1] Created socket
> with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -3
> (org.apache.kafka.common.network.Selector)
> [2020-07-08 18:48:09,457] DEBUG Set SASL client state to
> RECEIVE_APIVERSIONS_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,461] DEBUG Set SASL client state to
> SEND_HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,461] DEBUG Set SASL client state to
> RECEIVE_HANDSHAKE_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,461] DEBUG [NetworkClient clientId=admin-1] Completed
> connection to node -3. Fetching API versions.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] DEBUG [NetworkClient clientId=admin-1]
> Initialize connection to node broker2:2345 (id: -2 rack: null) for sending
> metadata request (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] DEBUG [NetworkClient clientId=admin-1]
> Initiating connection to node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] DEBUG Set SASL client state to
> SEND_APIVERSIONS_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,462] DEBUG Creating SaslClient:
> client=null;service=kafka;serviceHostname=broker2;mechs=[SCRAM-SHA-512]
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,463] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> SEND_CLIENT_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,464] DEBUG Set SASL client state to
> SEND_HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,464] DEBUG Set SASL client state to
> RECEIVE_HANDSHAKE_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,464] DEBUG Set SASL client state to INITIAL
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,471] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,471] DEBUG Set SASL client state to INTERMEDIATE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,472] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,472] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,472] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker1:2345 (id: -1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,472] DEBUG Set SASL client state to INITIAL
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,472] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,473] DEBUG Set SASL client state to INTERMEDIATE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,473] DEBUG Added sensor with name node--2.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,473] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,473] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] DEBUG Added sensor with name
> node--2.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,475] DEBUG Added sensor with name node--2.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,475] TRACE Registered metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,475] TRACE Registered metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,475] DEBUG [Producer clientId=admin-1] Created socket
> with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2
> (org.apache.kafka.common.network.Selector)
> [2020-07-08 18:48:09,475] DEBUG Set SASL client state to
> RECEIVE_APIVERSIONS_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,547] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FINAL_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,548] DEBUG [NetworkClient clientId=admin-1] Completed
> connection to node -2. Fetching API versions.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,548] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,548] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,548] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker1:2345 (id: -1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,583] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FINAL_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,584] DEBUG Set SASL client state to
> SEND_HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,584] DEBUG Set SASL client state to
> RECEIVE_HANDSHAKE_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,586] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,586] DEBUG Set SASL client state to COMPLETE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,587] DEBUG [NetworkClient clientId=admin-1]
> Initiating API versions fetch from node -1.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,587] TRACE [NetworkClient clientId=admin-1] No
> version information found when sending API_VERSIONS with correlation id 1
> to node -1. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,587] TRACE [NetworkClient clientId=admin-1] Sending
> API_VERSIONS {} with correlation id 1 to node -1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,589] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,589] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,590] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,590] DEBUG Set SASL client state to COMPLETE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,590] DEBUG Set SASL client state to INITIAL
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,591] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,591] DEBUG Set SASL client state to INTERMEDIATE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,591] DEBUG [NetworkClient clientId=admin-1]
> Initiating API versions fetch from node -3.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,591] TRACE [NetworkClient clientId=admin-1] No
> version information found when sending API_VERSIONS with correlation id 2
> to node -3. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,592] TRACE [NetworkClient clientId=admin-1] Sending
> API_VERSIONS {} with correlation id 2 to node -3
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,592] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,592] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,593] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -1 for API_VERSIONS with correlation id 1, received
> {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,595] DEBUG [NetworkClient clientId=admin-1] Recorded
> API versions for node -1: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8
> [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6
> [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0
> [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7):
> 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0
> to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0
> to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2
> [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2
> [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1
> [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3
> [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to
> 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1],
> OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to
> 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1
> [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to
> 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1
> [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to
> 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1],
> AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1
> [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0
> to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1],
> RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0
> to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1],
> DeleteGroups(42): 0 to 1 [usable: 1])
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,596] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,596] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,596] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,615] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FINAL_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,616] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -3 for API_VERSIONS with correlation id 2, received
> {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,617] DEBUG [NetworkClient clientId=admin-1] Recorded
> API versions for node -3: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8
> [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6
> [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0
> [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7):
> 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0
> to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0
> to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2
> [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2
> [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1
> [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3
> [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to
> 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1],
> OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to
> 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1
> [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to
> 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1
> [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to
> 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1],
> AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1
> [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0
> to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1],
> RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0
> to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1],
> DeleteGroups(42): 0 to 1 [usable: 1])
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,617] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,617] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,617] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,620] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,620] DEBUG Set SASL client state to COMPLETE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,620] DEBUG [NetworkClient clientId=admin-1]
> Initiating API versions fetch from node -2.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,620] TRACE [NetworkClient clientId=admin-1] No
> version information found when sending API_VERSIONS with correlation id 3
> to node -2. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,620] TRACE [NetworkClient clientId=admin-1] Sending
> API_VERSIONS {} with correlation id 3 to node -2
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,621] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,621] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,621] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker3:2345 (id: -3 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,621] DEBUG [NetworkClient clientId=admin-1] Sending
> metadata request (type=MetadataRequest, topics=) to node broker3:2345 (id:
> -3 rack: null) (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,622] TRACE [NetworkClient clientId=admin-1] Sending
> METADATA {topics=[],allow_auto_topic_creation=true} with correlation id 4
> to node -3 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,623] TRACE [NetworkClient clientId=admin-1] Sending
> FIND_COORDINATOR
> {coordinator_key=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,coordinator_type=0}
> with correlation id 0 to node -1 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,624] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -3 for METADATA with correlation id 4, received
> {throttle_time_ms=0,brokers=[{node_id=2,host=broker2,port=3411,rack=null},{node_id=1,host=broker1,port=3411,rack=null},{node_id=3,host=broker3,port=3411,rack=null}],cluster_id=EgYhY5ofQaOMSa3UOR-t_w,controller_id=2,topic_metadata=[]}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,630] INFO Cluster ID: EgYhY5ofQaOMSa3UOR-t_w
> (org.apache.kafka.clients.Metadata)
> [2020-07-08 18:48:09,631] DEBUG Updated cluster metadata version 2 to
> Cluster(id = EgYhY5ofQaOMSa3UOR-t_w, nodes = [broker1:2345 (id: 1 rack:
> null), broker3:2345 (id: 3 rack: null), broker2:2345 (id: 2 rack: null)],
> partitions = [], controller = broker2:2345 (id: 2 rack: null))
> (org.apache.kafka.clients.Metadata)
> [2020-07-08 18:48:09,632] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -1 for FIND_COORDINATOR with correlation id 0, received
> {throttle_time_ms=0,error_code=0,error_message=null,coordinator={node_id=1,host=broker1,port=3411}}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,633] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -2 for API_VERSIONS with correlation id 3, received
> {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,635] DEBUG [NetworkClient clientId=admin-1] Recorded
> API versions for node -2: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8
> [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6
> [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0
> [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7):
> 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0
> to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0
> to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2
> [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2
> [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1
> [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3
> [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to
> 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1],
> OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to
> 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1
> [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to
> 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1
> [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to
> 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1],
> AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1
> [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0
> to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1],
> RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0
> to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1],
> DeleteGroups(42): 0 to 1 [usable: 1])
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,636] DEBUG [NetworkClient clientId=admin-1]
> Initiating connection to node broker1:2345 (id: 1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,637] DEBUG Set SASL client state to
> SEND_APIVERSIONS_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,637] DEBUG Creating SaslClient:
> client=null;service=kafka;serviceHostname=broker1;mechs=[SCRAM-SHA-512]
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,638] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> SEND_CLIENT_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,639] DEBUG Added sensor with name node-1.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,639] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,639] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,639] TRACE Registered metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,639] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] DEBUG Added sensor with name
> node-1.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] DEBUG Added sensor with name node-1.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,641] TRACE Registered metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,641] TRACE Registered metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,641] DEBUG [Producer clientId=admin-1] Created socket
> with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1
> (org.apache.kafka.common.network.Selector)
> [2020-07-08 18:48:09,641] DEBUG Set SASL client state to
> RECEIVE_APIVERSIONS_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,641] DEBUG [NetworkClient clientId=admin-1] Completed
> connection to node 1. Fetching API versions.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,642] DEBUG Set SASL client state to
> SEND_HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,642] DEBUG Set SASL client state to
> RECEIVE_HANDSHAKE_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,643] DEBUG Set SASL client state to INITIAL
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,644] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,644] DEBUG Set SASL client state to INTERMEDIATE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,664] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FINAL_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,665] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,666] DEBUG Set SASL client state to COMPLETE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,666] DEBUG [NetworkClient clientId=admin-1]
> Initiating API versions fetch from node 1.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,666] TRACE [NetworkClient clientId=admin-1] No
> version information found when sending API_VERSIONS with correlation id 6
> to node 1. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,666] TRACE [NetworkClient clientId=admin-1] Sending
> API_VERSIONS {} with correlation id 6 to node 1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,668] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node 1 for API_VERSIONS with correlation id 6, received
> {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,669] DEBUG [NetworkClient clientId=admin-1] Recorded
> API versions for node 1: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8
> [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6
> [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0
> [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7):
> 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0
> to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0
> to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2
> [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2
> [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1
> [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3
> [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to
> 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1],
> OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to
> 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1
> [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to
> 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1
> [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to
> 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1],
> AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1
> [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0
> to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1],
> RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0
> to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1],
> DeleteGroups(42): 0 to 1 [usable: 1])
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,669] TRACE [NetworkClient clientId=admin-1] Sending
> DESCRIBE_GROUPS
> {group_ids=[spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0]}
> with correlation id 5 to node 1 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,671] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node 1 for DESCRIBE_GROUPS with correlation id 5, received
> {throttle_time_ms=0,groups=[{error_code=0,group_id=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,state=Stable,protocol_type=consumer,protocol=range,members=[{member_id=consumer-1-81b2f357-e145-4806-b987-12122f4072f6,client_id=consumer-1,client_host=/
> 10.0.0.1,member_metadata=java.nio.HeapByteBuffer[pos=0 lim=49
> cap=110],member_assignment=java.nio.HeapByteBuffer[pos=0 lim=57
> cap=57]}]}]} (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,683] TRACE [NetworkClient clientId=admin-1] Sending
> FIND_COORDINATOR
> {coordinator_key=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,coordinator_type=0}
> with correlation id 7 to node -1 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,685] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -1 for FIND_COORDINATOR with correlation id 7, received
> {throttle_time_ms=0,error_code=0,error_message=null,coordinator={node_id=1,host=broker1,port=3411}}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,686] TRACE [NetworkClient clientId=admin-1] Sending
> OFFSET_FETCH
> {group_id=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,topics=null}
> with correlation id 8 to node 1 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,689] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node 1 for OFFSET_FETCH with correlation id 8, received
> {throttle_time_ms=0,responses=[],error_code=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-count, group=admin-metrics, description=The current number
> of active connections., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-close-total, group=admin-metrics, description=The total
> number of connections closed, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-close-rate, group=admin-metrics, description=The number of
> connections closed per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] DEBUG Removed sensor with name
> connections-closed: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-creation-total, group=admin-metrics, description=The total
> number of new connections established, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-creation-rate, group=admin-metrics, description=The number
> of new connections established per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] DEBUG Removed sensor with name
> connections-created: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=successful-authentication-total, group=admin-metrics, description=The
> total number of connections with successful authentication, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=successful-authentication-rate, group=admin-metrics, description=The
> number of connections with successful authentication per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] DEBUG Removed sensor with name
> successful-authentication: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=failed-authentication-total, group=admin-metrics, description=The
> total number of connections with failed authentication, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=failed-authentication-rate, group=admin-metrics, description=The
> number of connections with failed authentication per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] DEBUG Removed sensor with name
> failed-authentication: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=network-io-total, group=admin-metrics, description=The total number
> of network operations (reads or writes) on all connections, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=network-io-rate, group=admin-metrics, description=The number of
> network operations (reads or writes) on all connections per second,
> tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] DEBUG Removed sensor with name
> bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-metrics, description=The total
> number of outgoing bytes sent to all servers, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-metrics, description=The number of
> outgoing bytes sent to all servers per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=request-total, group=admin-metrics, description=The total number of
> requests sent, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-metrics, description=The number of requests
> sent per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-metrics, description=The average size
> of requests sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-metrics, description=The maximum size
> of any request sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] DEBUG Removed sensor with name bytes-sent:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-metrics, description=The total
> number of bytes read off all sockets, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-metrics, description=The number of
> bytes read off all sockets per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=response-total, group=admin-metrics, description=The total number of
> responses received, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-metrics, description=The number of
> responses received per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] DEBUG Removed sensor with name bytes-received:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=select-total, group=admin-metrics, description=The total number of
> times the I/O layer checked for new I/O to perform, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=select-rate, group=admin-metrics, description=The number of times the
> I/O layer checked for new I/O to perform per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=io-wait-time-ns-avg, group=admin-metrics, description=The average
> length of time the I/O thread spent waiting for a socket ready for reads or
> writes in nanoseconds., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=io-waittime-total, group=admin-metrics, description=The total time
> the I/O thread spent waiting, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=io-wait-ratio, group=admin-metrics, description=The fraction of time
> the I/O thread spent waiting, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] DEBUG Removed sensor with name select-time:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=io-time-ns-avg, group=admin-metrics, description=The average length
> of time for I/O per select call in nanoseconds., tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=iotime-total, group=admin-metrics, description=The total time the I/O
> thread spent doing I/O, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=io-ratio, group=admin-metrics, description=The fraction of time the
> I/O thread spent doing I/O, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] DEBUG Removed sensor with name io-time:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] DEBUG Removed sensor with name
> node--1.bytes-sent (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] DEBUG Removed sensor with name
> node--1.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] DEBUG Removed sensor with name node--1.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] DEBUG Removed sensor with name
> node--3.bytes-sent (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] DEBUG Removed sensor with name
> node--3.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] DEBUG Removed sensor with name node--3.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] DEBUG Removed sensor with name
> node--2.bytes-sent (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] DEBUG Removed sensor with name
> node--2.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] DEBUG Removed sensor with name node--2.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,800] DEBUG Removed sensor with name node-1.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,800] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,800] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,800] TRACE Removed metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,807] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,807] DEBUG Removed sensor with name
> node-1.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,808] TRACE Removed metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,808] TRACE Removed metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,808] DEBUG Removed sensor with name node-1.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,808] TRACE LoginManager(serviceName=kafka,
> publicCredentials=[admin], refCount=0) released
> (org.apache.kafka.common.security.authenticator.LoginManager)
>
>
> Kindly check and let me know in case of any other details required.
>
> Regards,
> AnnPricksEdmund
>
> From: Ricardo Ferreira <ri...@riferrei.com>
> Date: Wednesday, 8 July 2020 at 6:29 PM
> To: "users@kafka.apache.org" <us...@kafka.apache.org>, Ann Pricks <
> pricks@dt.ae>
> Subject: Re: Consumer Groups Describe is not working
>
>
> Ann,
>
> You can try execute the CLI `kafka-consumer-groups` with TRACE enabled to
> dig a little deeper in the problem. In order to do this you need to:
>
> 1. Make a copy of your `$KAFKA_HOME/etc/kafka/tools-log4j.properties` file
>
> 2. Set `root.logger=TRACE,console`
>
> 3. Run `export
> KAFKA_OPTS="-Dlog4j.configuration=file:/tmp/tools-log4j.properties<file:///tmp/tools-log4j.properties>"`
> before executing the `kafka-consumer-groups` CLI.
>
> Thanks,
>
> -- Ricardo
> On 7/8/20 6:56 AM, Ann Pricks wrote:
>
> Hi Team,
>
>
>
> Any update on this.
>
>
>
>
>
> Regards,
>
> Pricks
>
>
>
> From: Ann Pricks <pr...@dt.ae>
>
> Date: Friday, 3 July 2020 at 4:10 PM
>
> To: "users@kafka.apache.org"<ma...@kafka.apache.org> <
> users@kafka.apache.org><ma...@kafka.apache.org>
>
> Subject: Consumer Groups Describe is not working
>
>
>
> Hi Team,
>
>
>
> Today, In our production cluster, we faced an issue with Kafka (Old
> offsets was getting pulled from spark streaming application) and couldn't
> debug the issue using kafka_consumer_group.sh CLI.
>
>
>
> Whenever we execute the below command to list the consumer groups, it is
> working fine. However, whenever we try to describe the consumer group to
> get to know the offset details, it didn't work (Nothing is getting
> displayed. Just blank).
>
>
>
> Command to list the consumer group (Working):
>
> /opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
>
> --bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
>
> --list \
>
> --command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
>
>
>
> Command to list the consumer group (Not Working):
>
> /opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
>
> --bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
>
> --describe \
>
> --group
> spark-kafka-source-f8e218d5-16d2-4e63-a25c-2f96fabb2809-605351645-driver-0 \
>
> --command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
>
>
>
> Jass Config File:
>
> sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
> required \
>
> username="admin" \
>
> password="123@admin";
>
> security.protocol=SASL_PLAINTEXT
>
> sasl.mechanism=SCRAM-SHA-512
>
> exclude.internal.topics=false
>
>
>
> Kindly help us to monitor our Kafka cluster in case of any issues.
>
>
>
> Details:
>
> Kafka Version: 2.0.0
>
> Security:
>
> sasl.enabled.mechanisms=SCRAM-SHA-512
>
> sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
>
> security.inter.broker.protocol=SASL_PLAINTEXT
>
>
>
> Please let us know in case of any other details required from our end.
>
>
>
> Regards,
>
> AnnPricksEdmund
>
Re: Consumer Groups Describe is not working
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Hi Ann,
It's common practice in many Spark Streaming apps to store offsets external
to Kafka. Especially when checkpointing is enabled.
Are you sure that the app is committing offsets to Kafka?
Kind regards,
Liam Clarke
On Thu, 9 Jul. 2020, 8:00 am Ann Pricks, <pr...@dt.ae> wrote:
> Hi Ricardo,
>
> Thanks for your kind response.
>
> As per your suggestion, I have enabled trace and PFB the content of the
> log file.
>
> Log File Content:
>
> [2020-07-08 18:48:08,963] INFO Registered kafka:type=kafka.Log4jController
> MBean (kafka.utils.Log4jControllerRegistration$)
> [2020-07-08 18:48:09,244] TRACE Registered metric named MetricName
> [name=count, group=kafka-metrics-count, description=total number of
> registered metrics, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,289] INFO Successfully logged in.
> (org.apache.kafka.common.security.authenticator.AbstractLogin)
> [2020-07-08 18:48:09,290] TRACE LoginManager(serviceName=kafka,
> publicCredentials=[admin], refCount=1) acquired
> (org.apache.kafka.common.security.authenticator.LoginManager)
> [2020-07-08 18:48:09,292] DEBUG Updated cluster metadata version 1 to
> Cluster(id = null, nodes = [broker1:2345 (id: -1 rack: null), broker2:2345
> (id: -2 rack: null), broker3:2345 (id: -3 rack: null)], partitions = [],
> controller = null) (org.apache.kafka.clients.Metadata)
> [2020-07-08 18:48:09,303] DEBUG Added sensor with name connections-closed:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,305] TRACE Registered metric named MetricName
> [name=connection-close-total, group=admin-metrics, description=The total
> number of connections closed, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=connection-close-rate, group=admin-metrics, description=The number of
> connections closed per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] DEBUG Added sensor with name
> connections-created: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=connection-creation-total, group=admin-metrics, description=The total
> number of new connections established, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=connection-creation-rate, group=admin-metrics, description=The number
> of new connections established per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] DEBUG Added sensor with name
> successful-authentication: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=successful-authentication-total, group=admin-metrics, description=The
> total number of connections with successful authentication, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=successful-authentication-rate, group=admin-metrics, description=The
> number of connections with successful authentication per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] DEBUG Added sensor with name
> failed-authentication: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,306] TRACE Registered metric named MetricName
> [name=failed-authentication-total, group=admin-metrics, description=The
> total number of connections with failed authentication, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] TRACE Registered metric named MetricName
> [name=failed-authentication-rate, group=admin-metrics, description=The
> number of connections with failed authentication per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] DEBUG Added sensor with name
> bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] TRACE Registered metric named MetricName
> [name=network-io-total, group=admin-metrics, description=The total number
> of network operations (reads or writes) on all connections, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] TRACE Registered metric named MetricName
> [name=network-io-rate, group=admin-metrics, description=The number of
> network operations (reads or writes) on all connections per second,
> tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] DEBUG Added sensor with name bytes-sent:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,307] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-metrics, description=The total
> number of outgoing bytes sent to all servers, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,308] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-metrics, description=The number of
> outgoing bytes sent to all servers per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,308] TRACE Registered metric named MetricName
> [name=request-total, group=admin-metrics, description=The total number of
> requests sent, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,308] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-metrics, description=The number of requests
> sent per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,308] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-metrics, description=The average size
> of requests sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-metrics, description=The maximum size
> of any request sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] DEBUG Added sensor with name bytes-received:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-metrics, description=The total
> number of bytes read off all sockets, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-metrics, description=The number of
> bytes read off all sockets per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=response-total, group=admin-metrics, description=The total number of
> responses received, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-metrics, description=The number of
> responses received per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] DEBUG Added sensor with name select-time:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,309] TRACE Registered metric named MetricName
> [name=select-total, group=admin-metrics, description=The total number of
> times the I/O layer checked for new I/O to perform, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=select-rate, group=admin-metrics, description=The number of times the
> I/O layer checked for new I/O to perform per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-wait-time-ns-avg, group=admin-metrics, description=The average
> length of time the I/O thread spent waiting for a socket ready for reads or
> writes in nanoseconds., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-waittime-total, group=admin-metrics, description=The total time
> the I/O thread spent waiting, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-wait-ratio, group=admin-metrics, description=The fraction of time
> the I/O thread spent waiting, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] DEBUG Added sensor with name io-time:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-time-ns-avg, group=admin-metrics, description=The average length
> of time for I/O per select call in nanoseconds., tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=iotime-total, group=admin-metrics, description=The total time the I/O
> thread spent doing I/O, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,311] TRACE Registered metric named MetricName
> [name=io-ratio, group=admin-metrics, description=The fraction of time the
> I/O thread spent doing I/O, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,312] TRACE Registered metric named MetricName
> [name=connection-count, group=admin-metrics, description=The current number
> of active connections., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,340] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,340] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,340] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker1:2345 (id: -1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,340] DEBUG [NetworkClient clientId=admin-1]
> Initialize connection to node broker1:2345 (id: -1 rack: null) for sending
> metadata request (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,340] DEBUG [NetworkClient clientId=admin-1]
> Initiating connection to node broker1:2345 (id: -1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,351] DEBUG Set SASL client state to
> SEND_APIVERSIONS_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,353] DEBUG Creating SaslClient:
> client=null;service=kafka;serviceHostname=broker1;mechs=[SCRAM-SHA-512]
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,375] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> SEND_CLIENT_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,378] DEBUG Added sensor with name node--1.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,379] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] DEBUG Added sensor with name
> node--1.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,383] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,384] DEBUG Added sensor with name node--1.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,384] TRACE Registered metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,384] TRACE Registered metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,385] DEBUG [Producer clientId=admin-1] Created socket
> with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
> (org.apache.kafka.common.network.Selector)
> [2020-07-08 18:48:09,452] DEBUG Set SASL client state to
> RECEIVE_APIVERSIONS_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,452] DEBUG [NetworkClient clientId=admin-1] Completed
> connection to node -1. Fetching API versions.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker3:2345 (id: -3 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] DEBUG [NetworkClient clientId=admin-1]
> Initialize connection to node broker3:2345 (id: -3 rack: null) for sending
> metadata request (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,453] DEBUG [NetworkClient clientId=admin-1]
> Initiating connection to node broker3:2345 (id: -3 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,454] DEBUG Set SASL client state to
> SEND_APIVERSIONS_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,454] DEBUG Creating SaslClient:
> client=null;service=kafka;serviceHostname=broker3;mechs=[SCRAM-SHA-512]
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,455] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> SEND_CLIENT_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,455] DEBUG Added sensor with name node--3.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,455] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,455] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,455] TRACE Registered metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] DEBUG Added sensor with name
> node--3.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,456] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,457] DEBUG Added sensor with name node--3.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,457] TRACE Registered metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,457] TRACE Registered metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,457] DEBUG [Producer clientId=admin-1] Created socket
> with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -3
> (org.apache.kafka.common.network.Selector)
> [2020-07-08 18:48:09,457] DEBUG Set SASL client state to
> RECEIVE_APIVERSIONS_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,461] DEBUG Set SASL client state to
> SEND_HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,461] DEBUG Set SASL client state to
> RECEIVE_HANDSHAKE_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,461] DEBUG [NetworkClient clientId=admin-1] Completed
> connection to node -3. Fetching API versions.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] DEBUG [NetworkClient clientId=admin-1]
> Initialize connection to node broker2:2345 (id: -2 rack: null) for sending
> metadata request (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] DEBUG [NetworkClient clientId=admin-1]
> Initiating connection to node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,462] DEBUG Set SASL client state to
> SEND_APIVERSIONS_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,462] DEBUG Creating SaslClient:
> client=null;service=kafka;serviceHostname=broker2;mechs=[SCRAM-SHA-512]
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,463] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> SEND_CLIENT_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,464] DEBUG Set SASL client state to
> SEND_HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,464] DEBUG Set SASL client state to
> RECEIVE_HANDSHAKE_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,464] DEBUG Set SASL client state to INITIAL
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,471] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,471] DEBUG Set SASL client state to INTERMEDIATE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,472] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,472] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,472] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker1:2345 (id: -1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,472] DEBUG Set SASL client state to INITIAL
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,472] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,473] DEBUG Set SASL client state to INTERMEDIATE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,473] DEBUG Added sensor with name node--2.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,473] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,473] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] DEBUG Added sensor with name
> node--2.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,474] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,475] DEBUG Added sensor with name node--2.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,475] TRACE Registered metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,475] TRACE Registered metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,475] DEBUG [Producer clientId=admin-1] Created socket
> with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2
> (org.apache.kafka.common.network.Selector)
> [2020-07-08 18:48:09,475] DEBUG Set SASL client state to
> RECEIVE_APIVERSIONS_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,547] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FINAL_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,548] DEBUG [NetworkClient clientId=admin-1] Completed
> connection to node -2. Fetching API versions.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,548] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,548] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,548] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker1:2345 (id: -1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,583] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FINAL_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,584] DEBUG Set SASL client state to
> SEND_HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,584] DEBUG Set SASL client state to
> RECEIVE_HANDSHAKE_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,586] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,586] DEBUG Set SASL client state to COMPLETE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,587] DEBUG [NetworkClient clientId=admin-1]
> Initiating API versions fetch from node -1.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,587] TRACE [NetworkClient clientId=admin-1] No
> version information found when sending API_VERSIONS with correlation id 1
> to node -1. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,587] TRACE [NetworkClient clientId=admin-1] Sending
> API_VERSIONS {} with correlation id 1 to node -1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,589] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,589] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,590] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,590] DEBUG Set SASL client state to COMPLETE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,590] DEBUG Set SASL client state to INITIAL
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,591] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,591] DEBUG Set SASL client state to INTERMEDIATE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,591] DEBUG [NetworkClient clientId=admin-1]
> Initiating API versions fetch from node -3.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,591] TRACE [NetworkClient clientId=admin-1] No
> version information found when sending API_VERSIONS with correlation id 2
> to node -3. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,592] TRACE [NetworkClient clientId=admin-1] Sending
> API_VERSIONS {} with correlation id 2 to node -3
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,592] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,592] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,593] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -1 for API_VERSIONS with correlation id 1, received
> {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,595] DEBUG [NetworkClient clientId=admin-1] Recorded
> API versions for node -1: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8
> [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6
> [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0
> [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7):
> 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0
> to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0
> to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2
> [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2
> [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1
> [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3
> [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to
> 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1],
> OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to
> 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1
> [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to
> 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1
> [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to
> 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1],
> AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1
> [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0
> to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1],
> RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0
> to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1],
> DeleteGroups(42): 0 to 1 [usable: 1])
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,596] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,596] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,596] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,615] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FINAL_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,616] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -3 for API_VERSIONS with correlation id 2, received
> {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,617] DEBUG [NetworkClient clientId=admin-1] Recorded
> API versions for node -3: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8
> [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6
> [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0
> [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7):
> 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0
> to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0
> to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2
> [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2
> [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1
> [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3
> [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to
> 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1],
> OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to
> 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1
> [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to
> 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1
> [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to
> 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1],
> AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1
> [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0
> to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1],
> RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0
> to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1],
> DeleteGroups(42): 0 to 1 [usable: 1])
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,617] TRACE [NetworkClient clientId=admin-1] Removing
> node broker3:2345 (id: -3 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,617] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,617] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker2:2345 (id: -2 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,620] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,620] DEBUG Set SASL client state to COMPLETE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,620] DEBUG [NetworkClient clientId=admin-1]
> Initiating API versions fetch from node -2.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,620] TRACE [NetworkClient clientId=admin-1] No
> version information found when sending API_VERSIONS with correlation id 3
> to node -2. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,620] TRACE [NetworkClient clientId=admin-1] Sending
> API_VERSIONS {} with correlation id 3 to node -2
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,621] TRACE [NetworkClient clientId=admin-1] Removing
> node broker1:2345 (id: -1 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 0
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,621] TRACE [NetworkClient clientId=admin-1] Removing
> node broker2:2345 (id: -2 rack: null) from least loaded node selection:
> is-blacked-out: false, in-flight-requests: 1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,621] TRACE [NetworkClient clientId=admin-1] Found
> least loaded node broker3:2345 (id: -3 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,621] DEBUG [NetworkClient clientId=admin-1] Sending
> metadata request (type=MetadataRequest, topics=) to node broker3:2345 (id:
> -3 rack: null) (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,622] TRACE [NetworkClient clientId=admin-1] Sending
> METADATA {topics=[],allow_auto_topic_creation=true} with correlation id 4
> to node -3 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,623] TRACE [NetworkClient clientId=admin-1] Sending
> FIND_COORDINATOR
> {coordinator_key=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,coordinator_type=0}
> with correlation id 0 to node -1 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,624] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -3 for METADATA with correlation id 4, received
> {throttle_time_ms=0,brokers=[{node_id=2,host=broker2,port=3411,rack=null},{node_id=1,host=broker1,port=3411,rack=null},{node_id=3,host=broker3,port=3411,rack=null}],cluster_id=EgYhY5ofQaOMSa3UOR-t_w,controller_id=2,topic_metadata=[]}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,630] INFO Cluster ID: EgYhY5ofQaOMSa3UOR-t_w
> (org.apache.kafka.clients.Metadata)
> [2020-07-08 18:48:09,631] DEBUG Updated cluster metadata version 2 to
> Cluster(id = EgYhY5ofQaOMSa3UOR-t_w, nodes = [broker1:2345 (id: 1 rack:
> null), broker3:2345 (id: 3 rack: null), broker2:2345 (id: 2 rack: null)],
> partitions = [], controller = broker2:2345 (id: 2 rack: null))
> (org.apache.kafka.clients.Metadata)
> [2020-07-08 18:48:09,632] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -1 for FIND_COORDINATOR with correlation id 0, received
> {throttle_time_ms=0,error_code=0,error_message=null,coordinator={node_id=1,host=broker1,port=3411}}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,633] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -2 for API_VERSIONS with correlation id 3, received
> {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,635] DEBUG [NetworkClient clientId=admin-1] Recorded
> API versions for node -2: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8
> [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6
> [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0
> [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7):
> 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0
> to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0
> to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2
> [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2
> [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1
> [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3
> [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to
> 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1],
> OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to
> 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1
> [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to
> 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1
> [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to
> 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1],
> AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1
> [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0
> to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1],
> RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0
> to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1],
> DeleteGroups(42): 0 to 1 [usable: 1])
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,636] DEBUG [NetworkClient clientId=admin-1]
> Initiating connection to node broker1:2345 (id: 1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,637] DEBUG Set SASL client state to
> SEND_APIVERSIONS_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,637] DEBUG Creating SaslClient:
> client=null;service=kafka;serviceHostname=broker1;mechs=[SCRAM-SHA-512]
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,638] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> SEND_CLIENT_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,639] DEBUG Added sensor with name node-1.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,639] TRACE Registered metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,639] TRACE Registered metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,639] TRACE Registered metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,639] TRACE Registered metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] DEBUG Added sensor with name
> node-1.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] TRACE Registered metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,640] DEBUG Added sensor with name node-1.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,641] TRACE Registered metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,641] TRACE Registered metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,641] DEBUG [Producer clientId=admin-1] Created socket
> with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1
> (org.apache.kafka.common.network.Selector)
> [2020-07-08 18:48:09,641] DEBUG Set SASL client state to
> RECEIVE_APIVERSIONS_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,641] DEBUG [NetworkClient clientId=admin-1] Completed
> connection to node 1. Fetching API versions.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,642] DEBUG Set SASL client state to
> SEND_HANDSHAKE_REQUEST
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,642] DEBUG Set SASL client state to
> RECEIVE_HANDSHAKE_RESPONSE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,643] DEBUG Set SASL client state to INITIAL
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,644] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FIRST_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,644] DEBUG Set SASL client state to INTERMEDIATE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,664] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> RECEIVE_SERVER_FINAL_MESSAGE
> (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,665] DEBUG Setting SASL/SCRAM_SHA_512 client state to
> COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
> [2020-07-08 18:48:09,666] DEBUG Set SASL client state to COMPLETE
> (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
> [2020-07-08 18:48:09,666] DEBUG [NetworkClient clientId=admin-1]
> Initiating API versions fetch from node 1.
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,666] TRACE [NetworkClient clientId=admin-1] No
> version information found when sending API_VERSIONS with correlation id 6
> to node 1. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,666] TRACE [NetworkClient clientId=admin-1] Sending
> API_VERSIONS {} with correlation id 6 to node 1
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,668] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node 1 for API_VERSIONS with correlation id 6, received
> {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,669] DEBUG [NetworkClient clientId=admin-1] Recorded
> API versions for node 1: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8
> [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6
> [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0
> [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7):
> 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0
> to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0
> to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2
> [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2
> [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1
> [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3
> [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to
> 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1],
> OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to
> 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1
> [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to
> 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1
> [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to
> 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1],
> AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1
> [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0
> to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1],
> RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0
> to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1],
> DeleteGroups(42): 0 to 1 [usable: 1])
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,669] TRACE [NetworkClient clientId=admin-1] Sending
> DESCRIBE_GROUPS
> {group_ids=[spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0]}
> with correlation id 5 to node 1 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,671] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node 1 for DESCRIBE_GROUPS with correlation id 5, received
> {throttle_time_ms=0,groups=[{error_code=0,group_id=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,state=Stable,protocol_type=consumer,protocol=range,members=[{member_id=consumer-1-81b2f357-e145-4806-b987-12122f4072f6,client_id=consumer-1,client_host=/
> 10.0.0.1,member_metadata=java.nio.HeapByteBuffer[pos=0 lim=49
> cap=110],member_assignment=java.nio.HeapByteBuffer[pos=0 lim=57
> cap=57]}]}]} (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,683] TRACE [NetworkClient clientId=admin-1] Sending
> FIND_COORDINATOR
> {coordinator_key=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,coordinator_type=0}
> with correlation id 7 to node -1 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,685] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node -1 for FIND_COORDINATOR with correlation id 7, received
> {throttle_time_ms=0,error_code=0,error_message=null,coordinator={node_id=1,host=broker1,port=3411}}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,686] TRACE [NetworkClient clientId=admin-1] Sending
> OFFSET_FETCH
> {group_id=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,topics=null}
> with correlation id 8 to node 1 (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,689] TRACE [NetworkClient clientId=admin-1] Completed
> receive from node 1 for OFFSET_FETCH with correlation id 8, received
> {throttle_time_ms=0,responses=[],error_code=0}
> (org.apache.kafka.clients.NetworkClient)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-count, group=admin-metrics, description=The current number
> of active connections., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-close-total, group=admin-metrics, description=The total
> number of connections closed, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-close-rate, group=admin-metrics, description=The number of
> connections closed per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] DEBUG Removed sensor with name
> connections-closed: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-creation-total, group=admin-metrics, description=The total
> number of new connections established, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=connection-creation-rate, group=admin-metrics, description=The number
> of new connections established per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] DEBUG Removed sensor with name
> connections-created: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=successful-authentication-total, group=admin-metrics, description=The
> total number of connections with successful authentication, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=successful-authentication-rate, group=admin-metrics, description=The
> number of connections with successful authentication per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] DEBUG Removed sensor with name
> successful-authentication: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,792] TRACE Removed metric named MetricName
> [name=failed-authentication-total, group=admin-metrics, description=The
> total number of connections with failed authentication, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=failed-authentication-rate, group=admin-metrics, description=The
> number of connections with failed authentication per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] DEBUG Removed sensor with name
> failed-authentication: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=network-io-total, group=admin-metrics, description=The total number
> of network operations (reads or writes) on all connections, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=network-io-rate, group=admin-metrics, description=The number of
> network operations (reads or writes) on all connections per second,
> tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] DEBUG Removed sensor with name
> bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-metrics, description=The total
> number of outgoing bytes sent to all servers, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-metrics, description=The number of
> outgoing bytes sent to all servers per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=request-total, group=admin-metrics, description=The total number of
> requests sent, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-metrics, description=The number of requests
> sent per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-metrics, description=The average size
> of requests sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-metrics, description=The maximum size
> of any request sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] DEBUG Removed sensor with name bytes-sent:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-metrics, description=The total
> number of bytes read off all sockets, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,793] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-metrics, description=The number of
> bytes read off all sockets per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=response-total, group=admin-metrics, description=The total number of
> responses received, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-metrics, description=The number of
> responses received per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] DEBUG Removed sensor with name bytes-received:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=select-total, group=admin-metrics, description=The total number of
> times the I/O layer checked for new I/O to perform, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=select-rate, group=admin-metrics, description=The number of times the
> I/O layer checked for new I/O to perform per second, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=io-wait-time-ns-avg, group=admin-metrics, description=The average
> length of time the I/O thread spent waiting for a socket ready for reads or
> writes in nanoseconds., tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=io-waittime-total, group=admin-metrics, description=The total time
> the I/O thread spent waiting, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] TRACE Removed metric named MetricName
> [name=io-wait-ratio, group=admin-metrics, description=The fraction of time
> the I/O thread spent waiting, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,794] DEBUG Removed sensor with name select-time:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=io-time-ns-avg, group=admin-metrics, description=The average length
> of time for I/O per select call in nanoseconds., tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=iotime-total, group=admin-metrics, description=The total time the I/O
> thread spent doing I/O, tags={}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=io-ratio, group=admin-metrics, description=The fraction of time the
> I/O thread spent doing I/O, tags={}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] DEBUG Removed sensor with name io-time:
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] DEBUG Removed sensor with name
> node--1.bytes-sent (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,795] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] DEBUG Removed sensor with name
> node--1.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] DEBUG Removed sensor with name node--1.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,796] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] DEBUG Removed sensor with name
> node--3.bytes-sent (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--3}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] DEBUG Removed sensor with name
> node--3.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] DEBUG Removed sensor with name node--3.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,797] TRACE Removed metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] DEBUG Removed sensor with name
> node--2.bytes-sent (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node--2}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] DEBUG Removed sensor with name
> node--2.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] TRACE Removed metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,798] DEBUG Removed sensor with name node--2.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=outgoing-byte-total, group=admin-node-metrics, description=The total
> number of outgoing bytes, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=outgoing-byte-rate, group=admin-node-metrics, description=The number
> of outgoing bytes per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=request-total, group=admin-node-metrics, description=The total number
> of requests sent, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=request-rate, group=admin-node-metrics, description=The number of
> requests sent per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=request-size-avg, group=admin-node-metrics, description=The average
> size of requests sent., tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,799] TRACE Removed metric named MetricName
> [name=request-size-max, group=admin-node-metrics, description=The maximum
> size of any request sent., tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,800] DEBUG Removed sensor with name node-1.bytes-sent
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,800] TRACE Removed metric named MetricName
> [name=incoming-byte-total, group=admin-node-metrics, description=The total
> number of incoming bytes, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,800] TRACE Removed metric named MetricName
> [name=incoming-byte-rate, group=admin-node-metrics, description=The number
> of incoming bytes per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,800] TRACE Removed metric named MetricName
> [name=response-total, group=admin-node-metrics, description=The total
> number of responses received, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,807] TRACE Removed metric named MetricName
> [name=response-rate, group=admin-node-metrics, description=The number of
> responses received per second, tags={node-id=node-1}]
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,807] DEBUG Removed sensor with name
> node-1.bytes-received (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,808] TRACE Removed metric named MetricName
> [name=request-latency-avg, group=admin-node-metrics, description=,
> tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,808] TRACE Removed metric named MetricName
> [name=request-latency-max, group=admin-node-metrics, description=,
> tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,808] DEBUG Removed sensor with name node-1.latency
> (org.apache.kafka.common.metrics.Metrics)
> [2020-07-08 18:48:09,808] TRACE LoginManager(serviceName=kafka,
> publicCredentials=[admin], refCount=0) released
> (org.apache.kafka.common.security.authenticator.LoginManager)
>
>
> Kindly check and let me know in case of any other details required.
>
> Regards,
> AnnPricksEdmund
>
> From: Ricardo Ferreira <ri...@riferrei.com>
> Date: Wednesday, 8 July 2020 at 6:29 PM
> To: "users@kafka.apache.org" <us...@kafka.apache.org>, Ann Pricks <
> pricks@dt.ae>
> Subject: Re: Consumer Groups Describe is not working
>
>
> Ann,
>
> You can try execute the CLI `kafka-consumer-groups` with TRACE enabled to
> dig a little deeper in the problem. In order to do this you need to:
>
> 1. Make a copy of your `$KAFKA_HOME/etc/kafka/tools-log4j.properties` file
>
> 2. Set `root.logger=TRACE,console`
>
> 3. Run `export
> KAFKA_OPTS="-Dlog4j.configuration=file:/tmp/tools-log4j.properties<file:///tmp/tools-log4j.properties>"`
> before executing the `kafka-consumer-groups` CLI.
>
> Thanks,
>
> -- Ricardo
> On 7/8/20 6:56 AM, Ann Pricks wrote:
>
> Hi Team,
>
>
>
> Any update on this.
>
>
>
>
>
> Regards,
>
> Pricks
>
>
>
> From: Ann Pricks <pr...@dt.ae>
>
> Date: Friday, 3 July 2020 at 4:10 PM
>
> To: "users@kafka.apache.org"<ma...@kafka.apache.org> <
> users@kafka.apache.org><ma...@kafka.apache.org>
>
> Subject: Consumer Groups Describe is not working
>
>
>
> Hi Team,
>
>
>
> Today, In our production cluster, we faced an issue with Kafka (Old
> offsets was getting pulled from spark streaming application) and couldn't
> debug the issue using kafka_consumer_group.sh CLI.
>
>
>
> Whenever we execute the below command to list the consumer groups, it is
> working fine. However, whenever we try to describe the consumer group to
> get to know the offset details, it didn't work (Nothing is getting
> displayed. Just blank).
>
>
>
> Command to list the consumer group (Working):
>
> /opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
>
> --bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
>
> --list \
>
> --command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
>
>
>
> Command to list the consumer group (Not Working):
>
> /opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
>
> --bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
>
> --describe \
>
> --group
> spark-kafka-source-f8e218d5-16d2-4e63-a25c-2f96fabb2809-605351645-driver-0 \
>
> --command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
>
>
>
> Jass Config File:
>
> sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule
> required \
>
> username="admin" \
>
> password="123@admin";
>
> security.protocol=SASL_PLAINTEXT
>
> sasl.mechanism=SCRAM-SHA-512
>
> exclude.internal.topics=false
>
>
>
> Kindly help us to monitor our Kafka cluster in case of any issues.
>
>
>
> Details:
>
> Kafka Version: 2.0.0
>
> Security:
>
> sasl.enabled.mechanisms=SCRAM-SHA-512
>
> sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
>
> security.inter.broker.protocol=SASL_PLAINTEXT
>
>
>
> Please let us know in case of any other details required from our end.
>
>
>
> Regards,
>
> AnnPricksEdmund
>
Re: Consumer Groups Describe is not working
Posted by Ann Pricks <pr...@dt.ae>.
Hi Ricardo,
Thanks for your kind response.
As per your suggestion, I have enabled trace and PFB the content of the log file.
Log File Content:
[2020-07-08 18:48:08,963] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-07-08 18:48:09,244] TRACE Registered metric named MetricName [name=count, group=kafka-metrics-count, description=total number of registered metrics, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,289] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin)
[2020-07-08 18:48:09,290] TRACE LoginManager(serviceName=kafka, publicCredentials=[admin], refCount=1) acquired (org.apache.kafka.common.security.authenticator.LoginManager)
[2020-07-08 18:48:09,292] DEBUG Updated cluster metadata version 1 to Cluster(id = null, nodes = [broker1:2345 (id: -1 rack: null), broker2:2345 (id: -2 rack: null), broker3:2345 (id: -3 rack: null)], partitions = [], controller = null) (org.apache.kafka.clients.Metadata)
[2020-07-08 18:48:09,303] DEBUG Added sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,305] TRACE Registered metric named MetricName [name=connection-close-total, group=admin-metrics, description=The total number of connections closed, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,306] TRACE Registered metric named MetricName [name=connection-close-rate, group=admin-metrics, description=The number of connections closed per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,306] DEBUG Added sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,306] TRACE Registered metric named MetricName [name=connection-creation-total, group=admin-metrics, description=The total number of new connections established, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,306] TRACE Registered metric named MetricName [name=connection-creation-rate, group=admin-metrics, description=The number of new connections established per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,306] DEBUG Added sensor with name successful-authentication: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,306] TRACE Registered metric named MetricName [name=successful-authentication-total, group=admin-metrics, description=The total number of connections with successful authentication, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,306] TRACE Registered metric named MetricName [name=successful-authentication-rate, group=admin-metrics, description=The number of connections with successful authentication per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,306] DEBUG Added sensor with name failed-authentication: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,306] TRACE Registered metric named MetricName [name=failed-authentication-total, group=admin-metrics, description=The total number of connections with failed authentication, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,307] TRACE Registered metric named MetricName [name=failed-authentication-rate, group=admin-metrics, description=The number of connections with failed authentication per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,307] DEBUG Added sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,307] TRACE Registered metric named MetricName [name=network-io-total, group=admin-metrics, description=The total number of network operations (reads or writes) on all connections, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,307] TRACE Registered metric named MetricName [name=network-io-rate, group=admin-metrics, description=The number of network operations (reads or writes) on all connections per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,307] DEBUG Added sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,307] TRACE Registered metric named MetricName [name=outgoing-byte-total, group=admin-metrics, description=The total number of outgoing bytes sent to all servers, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,308] TRACE Registered metric named MetricName [name=outgoing-byte-rate, group=admin-metrics, description=The number of outgoing bytes sent to all servers per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,308] TRACE Registered metric named MetricName [name=request-total, group=admin-metrics, description=The total number of requests sent, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,308] TRACE Registered metric named MetricName [name=request-rate, group=admin-metrics, description=The number of requests sent per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,308] TRACE Registered metric named MetricName [name=request-size-avg, group=admin-metrics, description=The average size of requests sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,309] TRACE Registered metric named MetricName [name=request-size-max, group=admin-metrics, description=The maximum size of any request sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,309] DEBUG Added sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,309] TRACE Registered metric named MetricName [name=incoming-byte-total, group=admin-metrics, description=The total number of bytes read off all sockets, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,309] TRACE Registered metric named MetricName [name=incoming-byte-rate, group=admin-metrics, description=The number of bytes read off all sockets per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,309] TRACE Registered metric named MetricName [name=response-total, group=admin-metrics, description=The total number of responses received, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,309] TRACE Registered metric named MetricName [name=response-rate, group=admin-metrics, description=The number of responses received per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,309] DEBUG Added sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,309] TRACE Registered metric named MetricName [name=select-total, group=admin-metrics, description=The total number of times the I/O layer checked for new I/O to perform, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,311] TRACE Registered metric named MetricName [name=select-rate, group=admin-metrics, description=The number of times the I/O layer checked for new I/O to perform per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,311] TRACE Registered metric named MetricName [name=io-wait-time-ns-avg, group=admin-metrics, description=The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,311] TRACE Registered metric named MetricName [name=io-waittime-total, group=admin-metrics, description=The total time the I/O thread spent waiting, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,311] TRACE Registered metric named MetricName [name=io-wait-ratio, group=admin-metrics, description=The fraction of time the I/O thread spent waiting, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,311] DEBUG Added sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,311] TRACE Registered metric named MetricName [name=io-time-ns-avg, group=admin-metrics, description=The average length of time for I/O per select call in nanoseconds., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,311] TRACE Registered metric named MetricName [name=iotime-total, group=admin-metrics, description=The total time the I/O thread spent doing I/O, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,311] TRACE Registered metric named MetricName [name=io-ratio, group=admin-metrics, description=The fraction of time the I/O thread spent doing I/O, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,312] TRACE Registered metric named MetricName [name=connection-count, group=admin-metrics, description=The current number of active connections., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,340] TRACE [NetworkClient clientId=admin-1] Removing node broker2:2345 (id: -2 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,340] TRACE [NetworkClient clientId=admin-1] Removing node broker3:2345 (id: -3 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,340] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker1:2345 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,340] DEBUG [NetworkClient clientId=admin-1] Initialize connection to node broker1:2345 (id: -1 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,340] DEBUG [NetworkClient clientId=admin-1] Initiating connection to node broker1:2345 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,351] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,353] DEBUG Creating SaslClient: client=null;service=kafka;serviceHostname=broker1;mechs=[SCRAM-SHA-512] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,375] DEBUG Setting SASL/SCRAM_SHA_512 client state to SEND_CLIENT_FIRST_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,378] DEBUG Added sensor with name node--1.bytes-sent (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,379] TRACE Registered metric named MetricName [name=outgoing-byte-total, group=admin-node-metrics, description=The total number of outgoing bytes, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,379] TRACE Registered metric named MetricName [name=outgoing-byte-rate, group=admin-node-metrics, description=The number of outgoing bytes per second, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,379] TRACE Registered metric named MetricName [name=request-total, group=admin-node-metrics, description=The total number of requests sent, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,379] TRACE Registered metric named MetricName [name=request-rate, group=admin-node-metrics, description=The number of requests sent per second, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,379] TRACE Registered metric named MetricName [name=request-size-avg, group=admin-node-metrics, description=The average size of requests sent., tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,383] TRACE Registered metric named MetricName [name=request-size-max, group=admin-node-metrics, description=The maximum size of any request sent., tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,383] DEBUG Added sensor with name node--1.bytes-received (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,383] TRACE Registered metric named MetricName [name=incoming-byte-total, group=admin-node-metrics, description=The total number of incoming bytes, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,383] TRACE Registered metric named MetricName [name=incoming-byte-rate, group=admin-node-metrics, description=The number of incoming bytes per second, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,383] TRACE Registered metric named MetricName [name=response-total, group=admin-node-metrics, description=The total number of responses received, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,383] TRACE Registered metric named MetricName [name=response-rate, group=admin-node-metrics, description=The number of responses received per second, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,384] DEBUG Added sensor with name node--1.latency (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,384] TRACE Registered metric named MetricName [name=request-latency-avg, group=admin-node-metrics, description=, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,384] TRACE Registered metric named MetricName [name=request-latency-max, group=admin-node-metrics, description=, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,385] DEBUG [Producer clientId=admin-1] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1 (org.apache.kafka.common.network.Selector)
[2020-07-08 18:48:09,452] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,452] DEBUG [NetworkClient clientId=admin-1] Completed connection to node -1. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,453] TRACE [NetworkClient clientId=admin-1] Removing node broker1:2345 (id: -1 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,453] TRACE [NetworkClient clientId=admin-1] Removing node broker2:2345 (id: -2 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,453] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker3:2345 (id: -3 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,453] DEBUG [NetworkClient clientId=admin-1] Initialize connection to node broker3:2345 (id: -3 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,453] DEBUG [NetworkClient clientId=admin-1] Initiating connection to node broker3:2345 (id: -3 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,454] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,454] DEBUG Creating SaslClient: client=null;service=kafka;serviceHostname=broker3;mechs=[SCRAM-SHA-512] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,455] DEBUG Setting SASL/SCRAM_SHA_512 client state to SEND_CLIENT_FIRST_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,455] DEBUG Added sensor with name node--3.bytes-sent (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,455] TRACE Registered metric named MetricName [name=outgoing-byte-total, group=admin-node-metrics, description=The total number of outgoing bytes, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,455] TRACE Registered metric named MetricName [name=outgoing-byte-rate, group=admin-node-metrics, description=The number of outgoing bytes per second, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,455] TRACE Registered metric named MetricName [name=request-total, group=admin-node-metrics, description=The total number of requests sent, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,456] TRACE Registered metric named MetricName [name=request-rate, group=admin-node-metrics, description=The number of requests sent per second, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,456] TRACE Registered metric named MetricName [name=request-size-avg, group=admin-node-metrics, description=The average size of requests sent., tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,456] TRACE Registered metric named MetricName [name=request-size-max, group=admin-node-metrics, description=The maximum size of any request sent., tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,456] DEBUG Added sensor with name node--3.bytes-received (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,456] TRACE Registered metric named MetricName [name=incoming-byte-total, group=admin-node-metrics, description=The total number of incoming bytes, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,456] TRACE Registered metric named MetricName [name=incoming-byte-rate, group=admin-node-metrics, description=The number of incoming bytes per second, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,456] TRACE Registered metric named MetricName [name=response-total, group=admin-node-metrics, description=The total number of responses received, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,456] TRACE Registered metric named MetricName [name=response-rate, group=admin-node-metrics, description=The number of responses received per second, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,457] DEBUG Added sensor with name node--3.latency (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,457] TRACE Registered metric named MetricName [name=request-latency-avg, group=admin-node-metrics, description=, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,457] TRACE Registered metric named MetricName [name=request-latency-max, group=admin-node-metrics, description=, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,457] DEBUG [Producer clientId=admin-1] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -3 (org.apache.kafka.common.network.Selector)
[2020-07-08 18:48:09,457] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,461] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,461] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,461] DEBUG [NetworkClient clientId=admin-1] Completed connection to node -3. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,462] TRACE [NetworkClient clientId=admin-1] Removing node broker3:2345 (id: -3 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,462] TRACE [NetworkClient clientId=admin-1] Removing node broker1:2345 (id: -1 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,462] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker2:2345 (id: -2 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,462] DEBUG [NetworkClient clientId=admin-1] Initialize connection to node broker2:2345 (id: -2 rack: null) for sending metadata request (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,462] DEBUG [NetworkClient clientId=admin-1] Initiating connection to node broker2:2345 (id: -2 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,462] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,462] DEBUG Creating SaslClient: client=null;service=kafka;serviceHostname=broker2;mechs=[SCRAM-SHA-512] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,463] DEBUG Setting SASL/SCRAM_SHA_512 client state to SEND_CLIENT_FIRST_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,464] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,464] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,464] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,471] DEBUG Setting SASL/SCRAM_SHA_512 client state to RECEIVE_SERVER_FIRST_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,471] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,472] TRACE [NetworkClient clientId=admin-1] Removing node broker2:2345 (id: -2 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,472] TRACE [NetworkClient clientId=admin-1] Removing node broker3:2345 (id: -3 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,472] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker1:2345 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,472] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,472] DEBUG Setting SASL/SCRAM_SHA_512 client state to RECEIVE_SERVER_FIRST_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,473] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,473] DEBUG Added sensor with name node--2.bytes-sent (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,473] TRACE Registered metric named MetricName [name=outgoing-byte-total, group=admin-node-metrics, description=The total number of outgoing bytes, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,473] TRACE Registered metric named MetricName [name=outgoing-byte-rate, group=admin-node-metrics, description=The number of outgoing bytes per second, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,474] TRACE Registered metric named MetricName [name=request-total, group=admin-node-metrics, description=The total number of requests sent, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,474] TRACE Registered metric named MetricName [name=request-rate, group=admin-node-metrics, description=The number of requests sent per second, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,474] TRACE Registered metric named MetricName [name=request-size-avg, group=admin-node-metrics, description=The average size of requests sent., tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,474] TRACE Registered metric named MetricName [name=request-size-max, group=admin-node-metrics, description=The maximum size of any request sent., tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,474] DEBUG Added sensor with name node--2.bytes-received (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,474] TRACE Registered metric named MetricName [name=incoming-byte-total, group=admin-node-metrics, description=The total number of incoming bytes, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,474] TRACE Registered metric named MetricName [name=incoming-byte-rate, group=admin-node-metrics, description=The number of incoming bytes per second, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,474] TRACE Registered metric named MetricName [name=response-total, group=admin-node-metrics, description=The total number of responses received, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,474] TRACE Registered metric named MetricName [name=response-rate, group=admin-node-metrics, description=The number of responses received per second, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,475] DEBUG Added sensor with name node--2.latency (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,475] TRACE Registered metric named MetricName [name=request-latency-avg, group=admin-node-metrics, description=, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,475] TRACE Registered metric named MetricName [name=request-latency-max, group=admin-node-metrics, description=, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,475] DEBUG [Producer clientId=admin-1] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -2 (org.apache.kafka.common.network.Selector)
[2020-07-08 18:48:09,475] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,547] DEBUG Setting SASL/SCRAM_SHA_512 client state to RECEIVE_SERVER_FINAL_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,548] DEBUG [NetworkClient clientId=admin-1] Completed connection to node -2. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,548] TRACE [NetworkClient clientId=admin-1] Removing node broker2:2345 (id: -2 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,548] TRACE [NetworkClient clientId=admin-1] Removing node broker3:2345 (id: -3 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,548] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker1:2345 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,583] DEBUG Setting SASL/SCRAM_SHA_512 client state to RECEIVE_SERVER_FINAL_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,584] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,584] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,586] DEBUG Setting SASL/SCRAM_SHA_512 client state to COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,586] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,587] DEBUG [NetworkClient clientId=admin-1] Initiating API versions fetch from node -1. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,587] TRACE [NetworkClient clientId=admin-1] No version information found when sending API_VERSIONS with correlation id 1 to node -1. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,587] TRACE [NetworkClient clientId=admin-1] Sending API_VERSIONS {} with correlation id 1 to node -1 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,589] TRACE [NetworkClient clientId=admin-1] Removing node broker3:2345 (id: -3 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,589] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker2:2345 (id: -2 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,590] DEBUG Setting SASL/SCRAM_SHA_512 client state to COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,590] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,590] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,591] DEBUG Setting SASL/SCRAM_SHA_512 client state to RECEIVE_SERVER_FIRST_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,591] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,591] DEBUG [NetworkClient clientId=admin-1] Initiating API versions fetch from node -3. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,591] TRACE [NetworkClient clientId=admin-1] No version information found when sending API_VERSIONS with correlation id 2 to node -3. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,592] TRACE [NetworkClient clientId=admin-1] Sending API_VERSIONS {} with correlation id 2 to node -3 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,592] TRACE [NetworkClient clientId=admin-1] Removing node broker1:2345 (id: -1 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 1 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,592] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker2:2345 (id: -2 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,593] TRACE [NetworkClient clientId=admin-1] Completed receive from node -1 for API_VERSIONS with correlation id 1, received {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0} (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,595] DEBUG [NetworkClient clientId=admin-1] Recorded API versions for node -1: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8 [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6 [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0 to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1]) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,596] TRACE [NetworkClient clientId=admin-1] Removing node broker3:2345 (id: -3 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 1 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,596] TRACE [NetworkClient clientId=admin-1] Removing node broker1:2345 (id: -1 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,596] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker2:2345 (id: -2 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,615] DEBUG Setting SASL/SCRAM_SHA_512 client state to RECEIVE_SERVER_FINAL_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,616] TRACE [NetworkClient clientId=admin-1] Completed receive from node -3 for API_VERSIONS with correlation id 2, received {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0} (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,617] DEBUG [NetworkClient clientId=admin-1] Recorded API versions for node -3: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8 [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6 [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0 to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1]) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,617] TRACE [NetworkClient clientId=admin-1] Removing node broker3:2345 (id: -3 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,617] TRACE [NetworkClient clientId=admin-1] Removing node broker1:2345 (id: -1 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,617] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker2:2345 (id: -2 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,620] DEBUG Setting SASL/SCRAM_SHA_512 client state to COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,620] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,620] DEBUG [NetworkClient clientId=admin-1] Initiating API versions fetch from node -2. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,620] TRACE [NetworkClient clientId=admin-1] No version information found when sending API_VERSIONS with correlation id 3 to node -2. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,620] TRACE [NetworkClient clientId=admin-1] Sending API_VERSIONS {} with correlation id 3 to node -2 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,621] TRACE [NetworkClient clientId=admin-1] Removing node broker1:2345 (id: -1 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,621] TRACE [NetworkClient clientId=admin-1] Removing node broker2:2345 (id: -2 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 1 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,621] TRACE [NetworkClient clientId=admin-1] Found least loaded node broker3:2345 (id: -3 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,621] DEBUG [NetworkClient clientId=admin-1] Sending metadata request (type=MetadataRequest, topics=) to node broker3:2345 (id: -3 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,622] TRACE [NetworkClient clientId=admin-1] Sending METADATA {topics=[],allow_auto_topic_creation=true} with correlation id 4 to node -3 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,623] TRACE [NetworkClient clientId=admin-1] Sending FIND_COORDINATOR {coordinator_key=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,coordinator_type=0} with correlation id 0 to node -1 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,624] TRACE [NetworkClient clientId=admin-1] Completed receive from node -3 for METADATA with correlation id 4, received {throttle_time_ms=0,brokers=[{node_id=2,host=broker2,port=3411,rack=null},{node_id=1,host=broker1,port=3411,rack=null},{node_id=3,host=broker3,port=3411,rack=null}],cluster_id=EgYhY5ofQaOMSa3UOR-t_w,controller_id=2,topic_metadata=[]} (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,630] INFO Cluster ID: EgYhY5ofQaOMSa3UOR-t_w (org.apache.kafka.clients.Metadata)
[2020-07-08 18:48:09,631] DEBUG Updated cluster metadata version 2 to Cluster(id = EgYhY5ofQaOMSa3UOR-t_w, nodes = [broker1:2345 (id: 1 rack: null), broker3:2345 (id: 3 rack: null), broker2:2345 (id: 2 rack: null)], partitions = [], controller = broker2:2345 (id: 2 rack: null)) (org.apache.kafka.clients.Metadata)
[2020-07-08 18:48:09,632] TRACE [NetworkClient clientId=admin-1] Completed receive from node -1 for FIND_COORDINATOR with correlation id 0, received {throttle_time_ms=0,error_code=0,error_message=null,coordinator={node_id=1,host=broker1,port=3411}} (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,633] TRACE [NetworkClient clientId=admin-1] Completed receive from node -2 for API_VERSIONS with correlation id 3, received {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0} (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,635] DEBUG [NetworkClient clientId=admin-1] Recorded API versions for node -2: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8 [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6 [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0 to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1]) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,636] DEBUG [NetworkClient clientId=admin-1] Initiating connection to node broker1:2345 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,637] DEBUG Set SASL client state to SEND_APIVERSIONS_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,637] DEBUG Creating SaslClient: client=null;service=kafka;serviceHostname=broker1;mechs=[SCRAM-SHA-512] (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,638] DEBUG Setting SASL/SCRAM_SHA_512 client state to SEND_CLIENT_FIRST_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,639] DEBUG Added sensor with name node-1.bytes-sent (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,639] TRACE Registered metric named MetricName [name=outgoing-byte-total, group=admin-node-metrics, description=The total number of outgoing bytes, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,639] TRACE Registered metric named MetricName [name=outgoing-byte-rate, group=admin-node-metrics, description=The number of outgoing bytes per second, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,639] TRACE Registered metric named MetricName [name=request-total, group=admin-node-metrics, description=The total number of requests sent, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,639] TRACE Registered metric named MetricName [name=request-rate, group=admin-node-metrics, description=The number of requests sent per second, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,640] TRACE Registered metric named MetricName [name=request-size-avg, group=admin-node-metrics, description=The average size of requests sent., tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,640] TRACE Registered metric named MetricName [name=request-size-max, group=admin-node-metrics, description=The maximum size of any request sent., tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,640] DEBUG Added sensor with name node-1.bytes-received (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,640] TRACE Registered metric named MetricName [name=incoming-byte-total, group=admin-node-metrics, description=The total number of incoming bytes, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,640] TRACE Registered metric named MetricName [name=incoming-byte-rate, group=admin-node-metrics, description=The number of incoming bytes per second, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,640] TRACE Registered metric named MetricName [name=response-total, group=admin-node-metrics, description=The total number of responses received, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,640] TRACE Registered metric named MetricName [name=response-rate, group=admin-node-metrics, description=The number of responses received per second, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,640] DEBUG Added sensor with name node-1.latency (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,641] TRACE Registered metric named MetricName [name=request-latency-avg, group=admin-node-metrics, description=, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,641] TRACE Registered metric named MetricName [name=request-latency-max, group=admin-node-metrics, description=, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,641] DEBUG [Producer clientId=admin-1] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1 (org.apache.kafka.common.network.Selector)
[2020-07-08 18:48:09,641] DEBUG Set SASL client state to RECEIVE_APIVERSIONS_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,641] DEBUG [NetworkClient clientId=admin-1] Completed connection to node 1. Fetching API versions. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,642] DEBUG Set SASL client state to SEND_HANDSHAKE_REQUEST (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,642] DEBUG Set SASL client state to RECEIVE_HANDSHAKE_RESPONSE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,643] DEBUG Set SASL client state to INITIAL (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,644] DEBUG Setting SASL/SCRAM_SHA_512 client state to RECEIVE_SERVER_FIRST_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,644] DEBUG Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,664] DEBUG Setting SASL/SCRAM_SHA_512 client state to RECEIVE_SERVER_FINAL_MESSAGE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,665] DEBUG Setting SASL/SCRAM_SHA_512 client state to COMPLETE (org.apache.kafka.common.security.scram.internals.ScramSaslClient)
[2020-07-08 18:48:09,666] DEBUG Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2020-07-08 18:48:09,666] DEBUG [NetworkClient clientId=admin-1] Initiating API versions fetch from node 1. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,666] TRACE [NetworkClient clientId=admin-1] No version information found when sending API_VERSIONS with correlation id 6 to node 1. Assuming version 2. (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,666] TRACE [NetworkClient clientId=admin-1] Sending API_VERSIONS {} with correlation id 6 to node 1 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,668] TRACE [NetworkClient clientId=admin-1] Completed receive from node 1 for API_VERSIONS with correlation id 6, received {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=6},{api_key=1,min_version=0,max_version=8},{api_key=2,min_version=0,max_version=3},{api_key=3,min_version=0,max_version=6},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=4},{api_key=9,min_version=0,max_version=4},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=3},{api_key=12,min_version=0,max_version=2},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=2},{api_key=15,min_version=0,max_version=2},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=2},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=1},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=1},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1}],throttle_time_ms=0} (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,669] DEBUG [NetworkClient clientId=admin-1] Recorded API versions for node 1: (Produce(0): 0 to 6 [usable: 6], Fetch(1): 0 to 8 [usable: 8], ListOffsets(2): 0 to 3 [usable: 3], Metadata(3): 0 to 6 [usable: 6], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 4 [usable: 4], OffsetFetch(9): 0 to 4 [usable: 4], FindCoordinator(10): 0 to 2 [usable: 2], JoinGroup(11): 0 to 3 [usable: 3], Heartbeat(12): 0 to 2 [usable: 2], LeaveGroup(13): 0 to 2 [usable: 2], SyncGroup(14): 0 to 2 [usable: 2], DescribeGroups(15): 0 to 2 [usable: 2], ListGroups(16): 0 to 2 [usable: 2], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 2 [usable: 2], CreateTopics(19): 0 to 3 [usable: 3], DeleteTopics(20): 0 to 2 [usable: 2], DeleteRecords(21): 0 to 1 [usable: 1], InitProducerId(22): 0 to 1 [usable: 1], OffsetForLeaderEpoch(23): 0 to 1 [usable: 1], AddPartitionsToTxn(24): 0 to 1 [usable: 1], AddOffsetsToTxn(25): 0 to 1 [usable: 1], EndTxn(26): 0 to 1 [usable: 1], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 1 [usable: 1], DescribeAcls(29): 0 to 1 [usable: 1], CreateAcls(30): 0 to 1 [usable: 1], DeleteAcls(31): 0 to 1 [usable: 1], DescribeConfigs(32): 0 to 2 [usable: 2], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 1 [usable: 1], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 to 1 [usable: 1], CreateDelegationToken(38): 0 to 1 [usable: 1], RenewDelegationToken(39): 0 to 1 [usable: 1], ExpireDelegationToken(40): 0 to 1 [usable: 1], DescribeDelegationToken(41): 0 to 1 [usable: 1], DeleteGroups(42): 0 to 1 [usable: 1]) (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,669] TRACE [NetworkClient clientId=admin-1] Sending DESCRIBE_GROUPS {group_ids=[spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0]} with correlation id 5 to node 1 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,671] TRACE [NetworkClient clientId=admin-1] Completed receive from node 1 for DESCRIBE_GROUPS with correlation id 5, received {throttle_time_ms=0,groups=[{error_code=0,group_id=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,state=Stable,protocol_type=consumer,protocol=range,members=[{member_id=consumer-1-81b2f357-e145-4806-b987-12122f4072f6,client_id=consumer-1,client_host=/10.0.0.1,member_metadata=java.nio.HeapByteBuffer[pos=0 lim=49 cap=110],member_assignment=java.nio.HeapByteBuffer[pos=0 lim=57 cap=57]}]}]} (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,683] TRACE [NetworkClient clientId=admin-1] Sending FIND_COORDINATOR {coordinator_key=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,coordinator_type=0} with correlation id 7 to node -1 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,685] TRACE [NetworkClient clientId=admin-1] Completed receive from node -1 for FIND_COORDINATOR with correlation id 7, received {throttle_time_ms=0,error_code=0,error_message=null,coordinator={node_id=1,host=broker1,port=3411}} (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,686] TRACE [NetworkClient clientId=admin-1] Sending OFFSET_FETCH {group_id=spark-kafka-source-2890f4c5-6119-48d4-8f1c-1d911ec4f36e--101837091-driver-0,topics=null} with correlation id 8 to node 1 (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,689] TRACE [NetworkClient clientId=admin-1] Completed receive from node 1 for OFFSET_FETCH with correlation id 8, received {throttle_time_ms=0,responses=[],error_code=0} (org.apache.kafka.clients.NetworkClient)
[2020-07-08 18:48:09,792] TRACE Removed metric named MetricName [name=connection-count, group=admin-metrics, description=The current number of active connections., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] TRACE Removed metric named MetricName [name=connection-close-total, group=admin-metrics, description=The total number of connections closed, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] TRACE Removed metric named MetricName [name=connection-close-rate, group=admin-metrics, description=The number of connections closed per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] DEBUG Removed sensor with name connections-closed: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] TRACE Removed metric named MetricName [name=connection-creation-total, group=admin-metrics, description=The total number of new connections established, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] TRACE Removed metric named MetricName [name=connection-creation-rate, group=admin-metrics, description=The number of new connections established per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] DEBUG Removed sensor with name connections-created: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] TRACE Removed metric named MetricName [name=successful-authentication-total, group=admin-metrics, description=The total number of connections with successful authentication, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] TRACE Removed metric named MetricName [name=successful-authentication-rate, group=admin-metrics, description=The number of connections with successful authentication per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] DEBUG Removed sensor with name successful-authentication: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,792] TRACE Removed metric named MetricName [name=failed-authentication-total, group=admin-metrics, description=The total number of connections with failed authentication, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=failed-authentication-rate, group=admin-metrics, description=The number of connections with failed authentication per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] DEBUG Removed sensor with name failed-authentication: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=network-io-total, group=admin-metrics, description=The total number of network operations (reads or writes) on all connections, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=network-io-rate, group=admin-metrics, description=The number of network operations (reads or writes) on all connections per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] DEBUG Removed sensor with name bytes-sent-received: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=outgoing-byte-total, group=admin-metrics, description=The total number of outgoing bytes sent to all servers, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=outgoing-byte-rate, group=admin-metrics, description=The number of outgoing bytes sent to all servers per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=request-total, group=admin-metrics, description=The total number of requests sent, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=request-rate, group=admin-metrics, description=The number of requests sent per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=request-size-avg, group=admin-metrics, description=The average size of requests sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=request-size-max, group=admin-metrics, description=The maximum size of any request sent., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] DEBUG Removed sensor with name bytes-sent: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=incoming-byte-total, group=admin-metrics, description=The total number of bytes read off all sockets, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,793] TRACE Removed metric named MetricName [name=incoming-byte-rate, group=admin-metrics, description=The number of bytes read off all sockets per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,794] TRACE Removed metric named MetricName [name=response-total, group=admin-metrics, description=The total number of responses received, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,794] TRACE Removed metric named MetricName [name=response-rate, group=admin-metrics, description=The number of responses received per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,794] DEBUG Removed sensor with name bytes-received: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,794] TRACE Removed metric named MetricName [name=select-total, group=admin-metrics, description=The total number of times the I/O layer checked for new I/O to perform, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,794] TRACE Removed metric named MetricName [name=select-rate, group=admin-metrics, description=The number of times the I/O layer checked for new I/O to perform per second, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,794] TRACE Removed metric named MetricName [name=io-wait-time-ns-avg, group=admin-metrics, description=The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,794] TRACE Removed metric named MetricName [name=io-waittime-total, group=admin-metrics, description=The total time the I/O thread spent waiting, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,794] TRACE Removed metric named MetricName [name=io-wait-ratio, group=admin-metrics, description=The fraction of time the I/O thread spent waiting, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,794] DEBUG Removed sensor with name select-time: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=io-time-ns-avg, group=admin-metrics, description=The average length of time for I/O per select call in nanoseconds., tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=iotime-total, group=admin-metrics, description=The total time the I/O thread spent doing I/O, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=io-ratio, group=admin-metrics, description=The fraction of time the I/O thread spent doing I/O, tags={}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] DEBUG Removed sensor with name io-time: (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=outgoing-byte-total, group=admin-node-metrics, description=The total number of outgoing bytes, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=outgoing-byte-rate, group=admin-node-metrics, description=The number of outgoing bytes per second, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=request-total, group=admin-node-metrics, description=The total number of requests sent, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=request-rate, group=admin-node-metrics, description=The number of requests sent per second, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=request-size-avg, group=admin-node-metrics, description=The average size of requests sent., tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=request-size-max, group=admin-node-metrics, description=The maximum size of any request sent., tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] DEBUG Removed sensor with name node--1.bytes-sent (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=incoming-byte-total, group=admin-node-metrics, description=The total number of incoming bytes, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,795] TRACE Removed metric named MetricName [name=incoming-byte-rate, group=admin-node-metrics, description=The number of incoming bytes per second, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=response-total, group=admin-node-metrics, description=The total number of responses received, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=response-rate, group=admin-node-metrics, description=The number of responses received per second, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] DEBUG Removed sensor with name node--1.bytes-received (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=request-latency-avg, group=admin-node-metrics, description=, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=request-latency-max, group=admin-node-metrics, description=, tags={node-id=node--1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] DEBUG Removed sensor with name node--1.latency (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=outgoing-byte-total, group=admin-node-metrics, description=The total number of outgoing bytes, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=outgoing-byte-rate, group=admin-node-metrics, description=The number of outgoing bytes per second, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=request-total, group=admin-node-metrics, description=The total number of requests sent, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=request-rate, group=admin-node-metrics, description=The number of requests sent per second, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=request-size-avg, group=admin-node-metrics, description=The average size of requests sent., tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,796] TRACE Removed metric named MetricName [name=request-size-max, group=admin-node-metrics, description=The maximum size of any request sent., tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] DEBUG Removed sensor with name node--3.bytes-sent (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] TRACE Removed metric named MetricName [name=incoming-byte-total, group=admin-node-metrics, description=The total number of incoming bytes, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] TRACE Removed metric named MetricName [name=incoming-byte-rate, group=admin-node-metrics, description=The number of incoming bytes per second, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] TRACE Removed metric named MetricName [name=response-total, group=admin-node-metrics, description=The total number of responses received, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] TRACE Removed metric named MetricName [name=response-rate, group=admin-node-metrics, description=The number of responses received per second, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] DEBUG Removed sensor with name node--3.bytes-received (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] TRACE Removed metric named MetricName [name=request-latency-avg, group=admin-node-metrics, description=, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] TRACE Removed metric named MetricName [name=request-latency-max, group=admin-node-metrics, description=, tags={node-id=node--3}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] DEBUG Removed sensor with name node--3.latency (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] TRACE Removed metric named MetricName [name=outgoing-byte-total, group=admin-node-metrics, description=The total number of outgoing bytes, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] TRACE Removed metric named MetricName [name=outgoing-byte-rate, group=admin-node-metrics, description=The number of outgoing bytes per second, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,797] TRACE Removed metric named MetricName [name=request-total, group=admin-node-metrics, description=The total number of requests sent, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] TRACE Removed metric named MetricName [name=request-rate, group=admin-node-metrics, description=The number of requests sent per second, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] TRACE Removed metric named MetricName [name=request-size-avg, group=admin-node-metrics, description=The average size of requests sent., tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] TRACE Removed metric named MetricName [name=request-size-max, group=admin-node-metrics, description=The maximum size of any request sent., tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] DEBUG Removed sensor with name node--2.bytes-sent (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] TRACE Removed metric named MetricName [name=incoming-byte-total, group=admin-node-metrics, description=The total number of incoming bytes, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] TRACE Removed metric named MetricName [name=incoming-byte-rate, group=admin-node-metrics, description=The number of incoming bytes per second, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] TRACE Removed metric named MetricName [name=response-total, group=admin-node-metrics, description=The total number of responses received, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] TRACE Removed metric named MetricName [name=response-rate, group=admin-node-metrics, description=The number of responses received per second, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] DEBUG Removed sensor with name node--2.bytes-received (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] TRACE Removed metric named MetricName [name=request-latency-avg, group=admin-node-metrics, description=, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] TRACE Removed metric named MetricName [name=request-latency-max, group=admin-node-metrics, description=, tags={node-id=node--2}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,798] DEBUG Removed sensor with name node--2.latency (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,799] TRACE Removed metric named MetricName [name=outgoing-byte-total, group=admin-node-metrics, description=The total number of outgoing bytes, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,799] TRACE Removed metric named MetricName [name=outgoing-byte-rate, group=admin-node-metrics, description=The number of outgoing bytes per second, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,799] TRACE Removed metric named MetricName [name=request-total, group=admin-node-metrics, description=The total number of requests sent, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,799] TRACE Removed metric named MetricName [name=request-rate, group=admin-node-metrics, description=The number of requests sent per second, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,799] TRACE Removed metric named MetricName [name=request-size-avg, group=admin-node-metrics, description=The average size of requests sent., tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,799] TRACE Removed metric named MetricName [name=request-size-max, group=admin-node-metrics, description=The maximum size of any request sent., tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,800] DEBUG Removed sensor with name node-1.bytes-sent (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,800] TRACE Removed metric named MetricName [name=incoming-byte-total, group=admin-node-metrics, description=The total number of incoming bytes, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,800] TRACE Removed metric named MetricName [name=incoming-byte-rate, group=admin-node-metrics, description=The number of incoming bytes per second, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,800] TRACE Removed metric named MetricName [name=response-total, group=admin-node-metrics, description=The total number of responses received, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,807] TRACE Removed metric named MetricName [name=response-rate, group=admin-node-metrics, description=The number of responses received per second, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,807] DEBUG Removed sensor with name node-1.bytes-received (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,808] TRACE Removed metric named MetricName [name=request-latency-avg, group=admin-node-metrics, description=, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,808] TRACE Removed metric named MetricName [name=request-latency-max, group=admin-node-metrics, description=, tags={node-id=node-1}] (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,808] DEBUG Removed sensor with name node-1.latency (org.apache.kafka.common.metrics.Metrics)
[2020-07-08 18:48:09,808] TRACE LoginManager(serviceName=kafka, publicCredentials=[admin], refCount=0) released (org.apache.kafka.common.security.authenticator.LoginManager)
Kindly check and let me know in case of any other details required.
Regards,
AnnPricksEdmund
From: Ricardo Ferreira <ri...@riferrei.com>
Date: Wednesday, 8 July 2020 at 6:29 PM
To: "users@kafka.apache.org" <us...@kafka.apache.org>, Ann Pricks <pr...@dt.ae>
Subject: Re: Consumer Groups Describe is not working
Ann,
You can try execute the CLI `kafka-consumer-groups` with TRACE enabled to dig a little deeper in the problem. In order to do this you need to:
1. Make a copy of your `$KAFKA_HOME/etc/kafka/tools-log4j.properties` file
2. Set `root.logger=TRACE,console`
3. Run `export KAFKA_OPTS="-Dlog4j.configuration=file:/tmp/tools-log4j.properties<file:///tmp/tools-log4j.properties>"` before executing the `kafka-consumer-groups` CLI.
Thanks,
-- Ricardo
On 7/8/20 6:56 AM, Ann Pricks wrote:
Hi Team,
Any update on this.
Regards,
Pricks
From: Ann Pricks <pr...@dt.ae>
Date: Friday, 3 July 2020 at 4:10 PM
To: "users@kafka.apache.org"<ma...@kafka.apache.org> <us...@kafka.apache.org>
Subject: Consumer Groups Describe is not working
Hi Team,
Today, In our production cluster, we faced an issue with Kafka (Old offsets was getting pulled from spark streaming application) and couldn't debug the issue using kafka_consumer_group.sh CLI.
Whenever we execute the below command to list the consumer groups, it is working fine. However, whenever we try to describe the consumer group to get to know the offset details, it didn't work (Nothing is getting displayed. Just blank).
Command to list the consumer group (Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--list \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Command to list the consumer group (Not Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--describe \
--group spark-kafka-source-f8e218d5-16d2-4e63-a25c-2f96fabb2809-605351645-driver-0 \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Jass Config File:
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="admin" \
password="123@admin";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
exclude.internal.topics=false
Kindly help us to monitor our Kafka cluster in case of any issues.
Details:
Kafka Version: 2.0.0
Security:
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
security.inter.broker.protocol=SASL_PLAINTEXT
Please let us know in case of any other details required from our end.
Regards,
AnnPricksEdmund
Re: Consumer Groups Describe is not working
Posted by Ann Pricks <pr...@dt.ae>.
Hi Ricardo,
Thanks for your kind response.
As per your suggestion, I have enabled trace and attached the log file.
Kindly check and let me know in case of any other details required.
Regards,
AnnPricksEdmund
From: Ricardo Ferreira <ri...@riferrei.com>
Date: Wednesday, 8 July 2020 at 6:29 PM
To: "users@kafka.apache.org" <us...@kafka.apache.org>, Ann Pricks <pr...@dt.ae>
Subject: Re: Consumer Groups Describe is not working
Ann,
You can try execute the CLI `kafka-consumer-groups` with TRACE enabled to dig a little deeper in the problem. In order to do this you need to:
1. Make a copy of your `$KAFKA_HOME/etc/kafka/tools-log4j.properties` file
2. Set `root.logger=TRACE,console`
3. Run `export KAFKA_OPTS="-Dlog4j.configuration=file:/tmp/tools-log4j.properties<file:///tmp/tools-log4j.properties>"` before executing the `kafka-consumer-groups` CLI.
Thanks,
-- Ricardo
On 7/8/20 6:56 AM, Ann Pricks wrote:
Hi Team,
Any update on this.
Regards,
Pricks
From: Ann Pricks <pr...@dt.ae>
Date: Friday, 3 July 2020 at 4:10 PM
To: "users@kafka.apache.org"<ma...@kafka.apache.org> <us...@kafka.apache.org>
Subject: Consumer Groups Describe is not working
Hi Team,
Today, In our production cluster, we faced an issue with Kafka (Old offsets was getting pulled from spark streaming application) and couldn't debug the issue using kafka_consumer_group.sh CLI.
Whenever we execute the below command to list the consumer groups, it is working fine. However, whenever we try to describe the consumer group to get to know the offset details, it didn't work (Nothing is getting displayed. Just blank).
Command to list the consumer group (Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--list \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Command to list the consumer group (Not Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--describe \
--group spark-kafka-source-f8e218d5-16d2-4e63-a25c-2f96fabb2809-605351645-driver-0 \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Jass Config File:
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="admin" \
password="123@admin";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
exclude.internal.topics=false
Kindly help us to monitor our Kafka cluster in case of any issues.
Details:
Kafka Version: 2.0.0
Security:
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
security.inter.broker.protocol=SASL_PLAINTEXT
Please let us know in case of any other details required from our end.
Regards,
AnnPricksEdmund
Re: Consumer Groups Describe is not working
Posted by Ricardo Ferreira <ri...@riferrei.com>.
Ann,
You can try execute the CLI `kafka-consumer-groups` with TRACE enabled
to dig a little deeper in the problem. In order to do this you need to:
1. Make a copy of your `$KAFKA_HOME/etc/kafka/tools-log4j.properties` file
2. Set `root.logger=TRACE,console`
3. Run `export
KAFKA_OPTS="-Dlog4j.configuration=file:/tmp/tools-log4j.properties"`
before executing the `kafka-consumer-groups` CLI.
Thanks,
-- Ricardo
On 7/8/20 6:56 AM, Ann Pricks wrote:
> Hi Team,
>
> Any update on this.
>
>
> Regards,
> Pricks
>
> From: Ann Pricks <pr...@dt.ae>
> Date: Friday, 3 July 2020 at 4:10 PM
> To: "users@kafka.apache.org" <us...@kafka.apache.org>
> Subject: Consumer Groups Describe is not working
>
> Hi Team,
>
> Today, In our production cluster, we faced an issue with Kafka (Old offsets was getting pulled from spark streaming application) and couldn't debug the issue using kafka_consumer_group.sh CLI.
>
> Whenever we execute the below command to list the consumer groups, it is working fine. However, whenever we try to describe the consumer group to get to know the offset details, it didn't work (Nothing is getting displayed. Just blank).
>
> Command to list the consumer group (Working):
> /opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
> --bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
> --list \
> --command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
>
> Command to list the consumer group (Not Working):
> /opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
> --bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
> --describe \
> --group spark-kafka-source-f8e218d5-16d2-4e63-a25c-2f96fabb2809-605351645-driver-0 \
> --command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
>
> Jass Config File:
> sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
> username="admin" \
> password="123@admin";
> security.protocol=SASL_PLAINTEXT
> sasl.mechanism=SCRAM-SHA-512
> exclude.internal.topics=false
>
> Kindly help us to monitor our Kafka cluster in case of any issues.
>
> Details:
> Kafka Version: 2.0.0
> Security:
> sasl.enabled.mechanisms=SCRAM-SHA-512
> sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
> security.inter.broker.protocol=SASL_PLAINTEXT
>
> Please let us know in case of any other details required from our end.
>
> Regards,
> AnnPricksEdmund
Re: Consumer Groups Describe is not working
Posted by Ma...@cognizant.com.
What error you aare getting . just make sure user have appropriate permission .
Please share the error if you are getting .
On 7/8/20, 3:56 AM, "Ann Pricks" <pr...@dt.ae> wrote:
[External]
Hi Team,
Any update on this.
Regards,
Pricks
From: Ann Pricks <pr...@dt.ae>
Date: Friday, 3 July 2020 at 4:10 PM
To: "users@kafka.apache.org" <us...@kafka.apache.org>
Subject: Consumer Groups Describe is not working
Hi Team,
Today, In our production cluster, we faced an issue with Kafka (Old offsets was getting pulled from spark streaming application) and couldn't debug the issue using kafka_consumer_group.sh CLI.
Whenever we execute the below command to list the consumer groups, it is working fine. However, whenever we try to describe the consumer group to get to know the offset details, it didn't work (Nothing is getting displayed. Just blank).
Command to list the consumer group (Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--list \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Command to list the consumer group (Not Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--describe \
--group spark-kafka-source-f8e218d5-16d2-4e63-a25c-2f96fabb2809-605351645-driver-0 \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Jass Config File:
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="admin" \
password="123@admin";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
exclude.internal.topics=false
Kindly help us to monitor our Kafka cluster in case of any issues.
Details:
Kafka Version: 2.0.0
Security:
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
security.inter.broker.protocol=SASL_PLAINTEXT
Please let us know in case of any other details required from our end.
Regards,
AnnPricksEdmund
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
Re: Consumer Groups Describe is not working
Posted by Ann Pricks <pr...@dt.ae>.
Hi Team,
Any update on this.
Regards,
Pricks
From: Ann Pricks <pr...@dt.ae>
Date: Friday, 3 July 2020 at 4:10 PM
To: "users@kafka.apache.org" <us...@kafka.apache.org>
Subject: Consumer Groups Describe is not working
Hi Team,
Today, In our production cluster, we faced an issue with Kafka (Old offsets was getting pulled from spark streaming application) and couldn't debug the issue using kafka_consumer_group.sh CLI.
Whenever we execute the below command to list the consumer groups, it is working fine. However, whenever we try to describe the consumer group to get to know the offset details, it didn't work (Nothing is getting displayed. Just blank).
Command to list the consumer group (Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--list \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Command to list the consumer group (Not Working):
/opt/kafka/kafka_2.11-2.0.0/bin/kafka-consumer-groups.sh \
--bootstrap-server broker1:2345,broker2:2345,broker3:2345 \
--describe \
--group spark-kafka-source-f8e218d5-16d2-4e63-a25c-2f96fabb2809-605351645-driver-0 \
--command-config /opt/kafka/kafka_2.11-2.0.0/config/jaas_config.conf
Jass Config File:
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="admin" \
password="123@admin";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
exclude.internal.topics=false
Kindly help us to monitor our Kafka cluster in case of any issues.
Details:
Kafka Version: 2.0.0
Security:
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
security.inter.broker.protocol=SASL_PLAINTEXT
Please let us know in case of any other details required from our end.
Regards,
AnnPricksEdmund