Re: Infinite loop trying to connect to coordinator

[Pierre Coquentin <pi...@gmail.com>]

FYI, since the upgrade to 0.11 two weeks ago, the error never occurred again

On Mon, Jul 17, 2017 at 9:28 AM, Pierre Coquentin <
pierre.coquentin@gmail.com> wrote:

> Is it possible than a max.poll.records too high can cause this instability?
>
> On Wed, Jul 12, 2017 at 8:43 AM, Pierre Coquentin <
> pierre.coquentin@gmail.com> wrote:
>
>> It was on our test environment and nothing was running when the incident
>> occurred.
>> In the server log we have a bunch of
>> [2017-07-11 11:52:15,330] WARN Attempting to send response via channel
>> for which there is no open connection, connection id 0
>> (kafka.network.Processor)
>> But the time doesn't match so I don't know if it's correlated or not
>>
>> On Tue, Jul 11, 2017 at 1:08 PM, John Yost <ho...@gmail.com> wrote:
>>
>>> Hi Pierre,
>>>
>>> Do your brokers remain responsive? In other words, do you see any other
>>> symptoms such as decreased write or read throughput which may indicate
>>> long
>>> GC pauses or possibly heavy load on your zookeeper cluster as evidenced
>>> by
>>> any SocketTimeoutExceptions on the Kafka and/or Zookeeper sides?
>>>
>>> --John
>>>
>>> On Tue, Jul 11, 2017 at 6:15 AM, Pierre Coquentin <
>>> pierre.coquentin@gmail.com> wrote:
>>>
>>> > Hi,
>>> >
>>> > We are using kafka 0.10.2 with 2 brokers and 2 application nodes
>>> composed
>>> > of 6 consumers each (all in one group). And recently we experienced
>>> > disconnection of both nodes simultaneously and an infinite retry to
>>> connect
>>> > to the coordinator. Currently, just restarting the nodes solve the
>>> problem
>>> > but it will occur a few hours later.
>>> > In the application log we see a lot of :
>>> > 11.07.2017 06:47:08,905 INFO
>>> > [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:631]
>>> > Marking the coordinator kafka-2:9092 (id: 2147483646 rack: null) dead
>>> for
>>> > group ABC
>>> > 11.07.2017 06:47:09,007 INFO
>>> > [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:586]
>>> > Discovered coordinator kafka-2:9092 (id: 2147483646 rack: null) for
>>> group
>>> > ABC.
>>> > 11.07.2017 06:47:09,008 INFO
>>> > [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:420]
>>> > (Re-)joining group ABC
>>> > 11.07.2017 06:47:09,274 INFO
>>> > [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:631]
>>> > Marking the coordinator kafka-2:9092 (id: 2147483646 rack: null) dead
>>> for
>>> > group ABC
>>> > 11.07.2017 06:47:09,375 INFO
>>> > [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:586]
>>> > Discovered coordinator kafka-2:9092 (id: 2147483646 rack: null) for
>>> group
>>> > ABC.
>>> > 11.07.2017 06:47:09,375 INFO
>>> > [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:420]
>>> > (Re-)joining group ABC
>>> > 11.07.2017 06:47:10,820 INFO
>>> > [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:631]
>>> > Marking the coordinator kafka-2:9092 (id: 2147483646 rack: null) dead
>>> for
>>> > group ABC
>>> > 11.07.2017 06:47:10,921 INFO
>>> > [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:586]
>>> > Discovered coordinator kafka-2:9092 (id: 2147483646 rack: null) for
>>> group
>>> > ABC.
>>> > 11.07.2017 06:47:10,922 INFO
>>> > [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:420]
>>> > (Re-)joining group ABC
>>> >
>>> > There is nothing in the log of the brokers.
>>> > We have no problem to contact the coordinator from both nodes. Could
>>> it be
>>> > a periodic instability of the network which leads to this infinite
>>> retries?
>>> > This problem could it be related to
>>> > https://issues.apache.org/jira/browse/KAFKA-5464 ?
>>> >
>>> > Here is the configuration of the Stream (lots of option are default
>>> ones)
>>> >         application.id = ABC
>>> >         application.server =
>>> >         bootstrap.servers = [kafka-1:9092, kafka-2:9092]
>>> >         buffered.records.per.partition = 1000
>>> >         cache.max.bytes.buffering = 10485760
>>> >         client.id =
>>> >         commit.interval.ms = 30000
>>> >         connections.max.idle.ms = 540000
>>> >         key.serde = class
>>> > org.apache.kafka.common.serialization.Serdes$StringSerde
>>> >         metadata.max.age.ms = 300000
>>> >         num.standby.replicas = 0
>>> >         num.stream.threads = 6
>>> >         partition.grouper = class
>>> > org.apache.kafka.streams.processor.DefaultPartitionGrouper
>>> >         poll.ms = 100
>>> >         receive.buffer.bytes = 32768
>>> >         reconnect.backoff.ms = 50
>>> >         replication.factor = 1
>>> >         request.timeout.ms = 40000
>>> >         retry.backoff.ms = 100
>>> >         rocksdb.config.setter = null
>>> >         security.protocol = PLAINTEXT
>>> >         send.buffer.bytes = 131072
>>> >         state.cleanup.delay.ms = 60000
>>> >         state.dir = null
>>> >         timestamp.extractor = class
>>> > org.apache.kafka.streams.processor.FailOnInvalidTimestamp
>>> >         value.serde = class com.sigfox.kafka.serde.AvroStr
>>> eamRecordSerde
>>> >         windowstore.changelog.additional.retention.ms = 86400000
>>> >         zookeeper.connect =
>>> > zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181
>>> >
>>> >
>>> > Any thoughts?
>>> > Regards,
>>> >
>>> > Pierre
>>> >
>>>
>>
>>
>