You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Boyang Chen (Jira)" <ji...@apache.org> on 2020/03/04 04:17:00 UTC
[jira] [Created] (KAFKA-9645) Records could not find corresponding
partition/task
Boyang Chen created KAFKA-9645:
----------------------------------
Summary: Records could not find corresponding partition/task
Key: KAFKA-9645
URL: https://issues.apache.org/jira/browse/KAFKA-9645
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.6.0
Reporter: Boyang Chen
Assignee: Boyang Chen
We could be hitting the illegal state when stream kicks off a rebalance with all tasks closed:
```
[2020-03-03T18:36:09-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,105] WARN [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Detected that the thread is being fenced. This implies that this thread missed a rebalance and dropped out of the consumer group. Will close out all assigned tasks and rejoin the consumer group. (org.apache.kafka.streams.processor.internals.StreamThread)
[2020-03-03T18:36:09-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,105] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Consumer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-restore-consumer, groupId=null] Subscribed to partition(s): stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000049-changelog-2 (org.apache.kafka.clients.consumer.KafkaConsumer)
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,286] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Producer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-1_1-producer, transactionalId=stream-soak-test-1_1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,287] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] task [1_1] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,287] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Consumer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions (org.apache.kafka.clients.consumer.KafkaConsumer)
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,290] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Producer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-3_2-producer, transactionalId=stream-soak-test-3_2] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,292] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] task [3_2] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,293] ERROR [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Unable to locate active task for received-record partition node-name-repartition-1. Current tasks: TaskManager
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) > MetadataState:
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) > Tasks:
(org.apache.kafka.streams.processor.internals.StreamThread)
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,293] ERROR [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Encountered the following exception during processing and the thread is going to shut down: (org.apache.kafka.streams.processor.internals.StreamThread)
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) java.lang.NullPointerException: Task was unexpectedly missing for partition node-name-repartition-1
at org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:984)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:820)
```
We should be more careful in this case by avoiding processing data when no corresponding task is found.
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:751)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:725)
[2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,294] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.processor.internals.StreamThread)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)