You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Levani Kokhreidze (Jira)" <ji...@apache.org> on 2020/03/03 08:18:00 UTC
[jira] [Updated] (KAFKA-9638) Do not trigger REBALANCING when
specific exceptions occur in Kafka Streams
[ https://issues.apache.org/jira/browse/KAFKA-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Levani Kokhreidze updated KAFKA-9638:
-------------------------------------
Description:
As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING of all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless, cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications.
In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify through some interface exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident.
was:
As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING on all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications.
In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify through some interface exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident.
> Do not trigger REBALANCING when specific exceptions occur in Kafka Streams
> ---------------------------------------------------------------------------
>
> Key: KAFKA-9638
> URL: https://issues.apache.org/jira/browse/KAFKA-9638
> Project: Kafka
> Issue Type: New Feature
> Reporter: Levani Kokhreidze
> Priority: Major
>
> As of now, when StreamThread encounters exception in Kafka Streams application, it will result in REBALANCING of all the tasks that were responsibility of the given thread. Problem with that is, if the exception was, lets say some logical exception, like NPE, REBALANCING is pretty much useless, cause all other threads will also die with the same NPE. This kind of mute rebalancing gives extra costs in terms of network traffic, IOPS, etc in case of large stateful applications.
> In addition, this behaviour causes global outage of the Kafka Streams application, instead of localized outage of the certain tasks. Would be great if Kafka Streams users could specify through some interface exceptions that must not trigger rebalancing of the tasks. StreamThread may still die, but in this case, we would have isolated incident.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)