You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Deepak Goyal (JIRA)" <ji...@apache.org> on 2018/05/31 19:09:00 UTC
[jira] [Comment Edited] (KAFKA-6976) Kafka Streams instances going
in to DEAD state
[ https://issues.apache.org/jira/browse/KAFKA-6976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497020#comment-16497020 ]
Deepak Goyal edited comment on KAFKA-6976 at 5/31/18 7:08 PM:
--------------------------------------------------------------
Furthermore, in the course of our application development, our application need consuming from four topics (each with 256 partitions). In this scenario, we could achieve a working cluster with only 60 machines. Increasing beyond 60 instances, all instances including the older ones went into DEAD state. We have an approximate need of 400 machines in our cluster.
was (Author: _deepakgoyal):
Furthermore, in the course of our application development, our application need consuming from four topics (each with 256 partitions). In this scenario, we could achieve a working cluster with only 60 machines. We have an approximate need of 400 machines in our cluster.
> Kafka Streams instances going in to DEAD state
> ----------------------------------------------
>
> Key: KAFKA-6976
> URL: https://issues.apache.org/jira/browse/KAFKA-6976
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 0.10.2.0
> Reporter: Deepak Goyal
> Priority: Blocker
>
> We are using Kafka 0.10.2.0, Kafka-Streams 1.1.0. We have Kafka Cluster of 16 machines, and topic that is being consumed by Kafka Streams has 256 partitions. We spawned 400 machines of Kakfa Streams application. We see that all of the StreamThreads go in to DEAD state.
> {quote}{{[2018-05-25 05:59:29,282] INFO stream-thread [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1] State transition from PENDING_SHUTDOWN to DEAD (org.apache.kafka.streams.processor.internals.StreamThread) [2018-05-25 05:59:29,282] INFO stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7] State transition from REBALANCING to ERROR (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] WARN stream-client [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7] All stream threads have died. The instance will be in error state and should be closed. (org.apache.kafka.streams.KafkaStreams) [2018-05-25 05:59:29,282] INFO stream-thread [ksapp-19f923d7-5f9e-4137-b79f-ee20945a7dd7-StreamThread-1] Shutdown complete (org.apache.kafka.streams.processor.internals.StreamThread)}}
> {quote}
> Please note that when we only have 100 kafka-streams application machines, things are working as expected. We see that instances are consuming messages from topic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)