You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "KUMAR (JIRA)" <ji...@apache.org> on 2019/07/31 02:10:00 UTC
[jira] [Created] (KAFKA-8737) TaskMigrated Exception while
rebalancing kafka streams
KUMAR created KAFKA-8737:
----------------------------
Summary: TaskMigrated Exception while rebalancing kafka streams
Key: KAFKA-8737
URL: https://issues.apache.org/jira/browse/KAFKA-8737
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 1.0.1, 1.0.0
Environment: 20 partitions
1 topic
8 Streamer service
topic-region-1 9 7841726 8236017 394291 streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/ streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
topic-region-1 15 7421710 7467666 45956 streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/ streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
topic-region-1 19 7737360 8120611 383251 streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/ streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
topic-region-1
Reporter: KUMAR
Kafka streams throws following exception while restart of a stream client service -
o.a.k.s.p.internals.StreamThread.? - stream-thread [streams-subscriberstopic-region-1-32d968e3-f892-4772-a7a4-6f684d7e43c9-StreamThread-1] Detected a task that got migrated to another thread. This implies that this thread missed a rebalance and dropped out of the consumer group. Trying to rejoin the consumer group now.
org.apache.kafka.streams.errors.TaskMigratedException: Log end offset of topic-region-1-12 should not change while restoring: old end offset 6286727, current offset 6380997
Kafka version is 1.0.0 and we have back merged the fix for KIP-6269-
[https://github.com/apache/kafka/pull/4300/files#|https://github.com/apache/kafka/pull/4300/files]
However we observe that there seems to be an issue in rebalance when "auto.offset.reset" is configured as "latest". Based on log analysis we see following behavior -
# StreamThread starts a restore consumer
# While Fetching it gets offset out of range o.a.k.c.consumer.internals.Fetcher.? - [Consumer clientId=streams-subscriberstopic-region-1-11b2d7fb-11ce-4b0b-a40a-388d3c7b6bc9-StreamThread-1-restore-
consumer, groupId=] Fetch READ_UNCOMMITTED at offset 246431 for partition topic-region-1-12 returned fetch data (error=OFFSET_OUT_OF_RANGE, highWaterMark=-1, lastStableOffset = -1, logStartOffset = -1,
abortedTransactions = null, recordsSizeInBytes=0)
# Fetcher tries to reset the offset
# While reset the offset it appears it is changing the offset position and causing TaskMigrated exception
Above test repeated with "auto.offset.reset" is configured as "earliest" does not throw any TaskMigrated exception as in earliest case we are not reseting the restore consumer position.
Please let us know if this is possible and if a fix would be needed for the offset reset piece when set to latest.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)