You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Boyang Chen (Jira)" <ji...@apache.org> on 2020/03/10 17:52:00 UTC
[jira] [Resolved] (KAFKA-9674) Task corruption should also close
the producer if necessary
[ https://issues.apache.org/jira/browse/KAFKA-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boyang Chen resolved KAFKA-9674.
--------------------------------
Resolution: Fixed
> Task corruption should also close the producer if necessary
> -----------------------------------------------------------
>
> Key: KAFKA-9674
> URL: https://issues.apache.org/jira/browse/KAFKA-9674
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Boyang Chen
> Assignee: Boyang Chen
> Priority: Major
>
> The task revive call only transits the task to CREATED mode. It should handle the recreation of task producer as well.
> Sequence is like:
> # Task hits out of range exception and throws CorruptedException
> # Task producer closed along with the task
> # Task revived and rebalance triggered
> # Task was assigned back to the same thread
> # Trying to use task producer will throw as it has already been closed.
> The full log:
>
> [2020-03-03T21:56:29-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:29,070] WARN [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] stream-thread [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] Encountered org.apache.kafka.clients.consumer.OffsetOutOfRangeException fetching records from restore consumer for partitions [stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000019-changelog-0], it is likely that the consumer's position has fallen out of the topic partition offset range because the topic was truncated or compacted on the broker, marking the corresponding tasks as corrupted and re-initializing it later. (org.apache.kafka.streams.processor.internals.StoreChangelogReader)
> [2020-03-03T21:56:29-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:29,071] WARN [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] stream-thread [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] Detected the states of tasks \{1_0=[stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000019-changelog-0]} are corrupted. Will close the task as dirty and re-create and bootstrap from scratch. (org.apache.kafka.streams.processor.internals.StreamThread)
>
> [2020-03-03T21:56:30-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:30,010] INFO [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] [Producer clientId=stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
>
>
> [2020-03-03T21:56:30-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:30,017] INFO [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] stream-thread [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] task [1_0] Closed clean (org.apache.kafka.streams.processor.internals.StreamTask)
>
>
> [2020-03-03T21:56:22-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:22,827] INFO [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] [Producer clientId=stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2020-03-03T21:56:22-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:22,829] INFO [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] stream-thread [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] task [1_0] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)