You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Matthias J. Sax (JIRA)" <ji...@apache.org> on 2018/02/05 19:56:00 UTC
[jira] [Updated] (KAFKA-6534) Consumer.poll may not trigger
rebalance in time when there is a task migration
[ https://issues.apache.org/jira/browse/KAFKA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias J. Sax updated KAFKA-6534:
-----------------------------------
Description:
When Streams detect a task migration event in one of its thread, today it will always let its trigger to call {{consumer.poll}} hoping it could trigger the rebalance and hence clean up the records buffered from the partitions that on longer owned. However, because the rebalance is based on heartbeat responses which has a window of race, the rebalance is not always guaranteed to be triggered when task migration happens. As a result it could cause the records buffered in consumer to not be cleaned up and later be processed by Streams, realizing it no longer belongs to the thread, causing:
{code:java}
java.lang.IllegalStateException: Record's partition does not belong to this partition-group.
{code}
was:
When Streams detect a task migration event in one of its thread, today it will always let its trigger to call {{consumer.poll}} hoping it could trigger the rebalance and hence clean up the records buffered from the partitions that on longer owned. However, because the rebalance is based on HB responses which has a window of race, the rebalance is not always guaranteed to be triggered when task migration happens. As a result it could cause the records buffered in consumer to not be cleaned up and later be processed by Streams, realizing it no longer belongs to the thread, causing:
{code}
java.lang.IllegalStateException: Record's partition does not belong to this partition-group.
{code}
> Consumer.poll may not trigger rebalance in time when there is a task migration
> ------------------------------------------------------------------------------
>
> Key: KAFKA-6534
> URL: https://issues.apache.org/jira/browse/KAFKA-6534
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Guozhang Wang
> Priority: Major
>
> When Streams detect a task migration event in one of its thread, today it will always let its trigger to call {{consumer.poll}} hoping it could trigger the rebalance and hence clean up the records buffered from the partitions that on longer owned. However, because the rebalance is based on heartbeat responses which has a window of race, the rebalance is not always guaranteed to be triggered when task migration happens. As a result it could cause the records buffered in consumer to not be cleaned up and later be processed by Streams, realizing it no longer belongs to the thread, causing:
> {code:java}
> java.lang.IllegalStateException: Record's partition does not belong to this partition-group.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)