You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Matthias J. Sax (Jira)" <ji...@apache.org> on 2020/05/15 02:51:00 UTC
[jira] [Resolved] (KAFKA-7133) DisconnectException every 5 minutes
in single restore consumer thread
[ https://issues.apache.org/jira/browse/KAFKA-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias J. Sax resolved KAFKA-7133.
------------------------------------
Resolution: Abandoned
> DisconnectException every 5 minutes in single restore consumer thread
> ---------------------------------------------------------------------
>
> Key: KAFKA-7133
> URL: https://issues.apache.org/jira/browse/KAFKA-7133
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 1.1.0
> Environment: Kafka Streams application in Kubernetes.
> Kafka Server in Docker on machine in host mode
> Reporter: Chris Schwarzfischer
> Priority: Major
>
> One of our streams applications (and only this one) gets a {{org.apache.kafka.common.errors.DisconnectException}} almost exactly every 5 minutes.
> The application has two of
> KStream -> KGroupedStream -> KTable -> KGroupedTable -> KTable
> aggregations.
> Relevant config is in Streams:
> {code:java}
> this.properties.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.AT_LEAST_ONCE);
> //...
> this.properties.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2);
> this.properties.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 1024 * 1024 * 500 /* 500 MB */ );
> this.properties.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 1024 * 1024 * 100 /* 100 MB */);
> this.properties.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, 1024 * 1024 * 50 /* 50 MB */);
> {code}
> On the broker:
> {noformat}
> KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
> KAFKA_OFFSETS_RETENTION_MINUTES: 108000
> KAFKA_MIN_INSYNC_REPLICAS: 2
> KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
> KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
> KAFKA_TRANSACTIONAL_ID_EXPIRATION_MS: 2147483000
> KAFKA_LOG_RETENTION_HOURS: 2688
> KAFKA_OFFSETS_RETENTION_CHECK_INTERVAL_MS: 1200000
> KAFKA_ZOOKEEPER_SESSION_TIMEOUT_MS: 12000
> {noformat}
> Logging gives us a single restore consumer thread that throws exceptions every 5 mins:
>
> {noformat}
> July 4th 2018, 15:38:51.560 dockertest03 2018-07-04T13:38:51,559Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to node 1: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:37:54.833 dockertest03 2018-07-04T13:37:54,832Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to node 3: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:37:54.833 dockertest03 2018-07-04T13:37:54,832Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to node 2: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:32:26.379 dockertest03 2018-07-04T13:32:26,378Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to node 1: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:32:01.926 dockertest03 2018-07-04T13:32:01,925Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to node 2: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:32:01.926 dockertest03 2018-07-04T13:32:01,925Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to node 3: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:26:53.886 dockertest03 2018-07-04T13:26:53,886Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to node 1: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:26:14.576 dockertest03 2018-07-04T13:26:14,575Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to node 3: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:26:14.575 dockertest03 2018-07-04T13:26:14,575Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to node 2: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:21:13.410 dockertest03 2018-07-04T13:21:13,409Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to node 1: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:20:39.618 dockertest03 2018-07-04T13:20:39,617Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to node 3: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:20:39.617 dockertest03 2018-07-04T13:20:39,617Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to node 2: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:15:31.454 dockertest03 2018-07-04T13:15:31,454Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to node 1: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:15:00.759 dockertest03 2018-07-04T13:15:00,758Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to node 3: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:15:00.758 dockertest03 2018-07-04T13:15:00,758Z INFO : [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: FetchSessionHandler::handleError:440 - [Consumer clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to node 2: org.apache.kafka.common.errors.DisconnectException.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)