You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Levani Kokhreidze (Jira)" <ji...@apache.org> on 2020/09/01 09:42:00 UTC
[jira] [Commented] (KAFKA-10454) Kafka Streams Stuck in infinite
REBALANCING loop when stream <> table join partitions don't match
[ https://issues.apache.org/jira/browse/KAFKA-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188313#comment-17188313 ]
Levani Kokhreidze commented on KAFKA-10454:
-------------------------------------------
CC [~mjsax] [~vvcephei] [~guozhang]
I may try to fix this as soon as I can - but if any chance somebody wants to pick it up before me, feel free to do so.
> Kafka Streams Stuck in infinite REBALANCING loop when stream <> table join partitions don't match
> -------------------------------------------------------------------------------------------------
>
> Key: KAFKA-10454
> URL: https://issues.apache.org/jira/browse/KAFKA-10454
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Levani Kokhreidze
> Priority: Major
> Fix For: 2.6.0
>
>
> Here's integration test: [https://github.com/apache/kafka/pull/9237]
>
> From the first glance, issue is that when one joins stream to table, and table source topic doesn't have same number of partitions as stream topic, `StateChangelogReader` tries to recover state from changelog (which in this case is the same as source topic) for table from partitions that don't exist. Logs are spammed with:
>
> {code:java}
> [2020-09-01 12:33:07,508] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,508] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,508] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,510] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,513] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,515] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,517] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,518] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,518] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,518] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,520] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,522] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,524] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,524] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,525] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,525] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,526] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,527] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,527] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,527] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-3 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,529] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-0 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,529] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-1 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716) [2020-09-01 12:33:07,529] INFO stream-thread [app-StreamTableJoinInfiniteLoopIntegrationTestloop-86ae06c3-5758-429f-9d29-94ed516db126-StreamThread-1] End offset for changelog topic-b-StreamTableJoinInfiniteLoopIntegrationTestloop-2 cannot be found; will retry in the next time. (org.apache.kafka.streams.processor.internals.StoreChangelogReader:716)
> {code}
> And Kafka Streams never moves to RUNNING state.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)