You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/03/04 04:25:00 UTC
[jira] [Commented] (KAFKA-9645) Records could not find
corresponding partition/task
[ https://issues.apache.org/jira/browse/KAFKA-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050887#comment-17050887 ]
ASF GitHub Bot commented on KAFKA-9645:
---------------------------------------
abbccdda commented on pull request #8220: KAFKA-9645: Remove Illegal State Check for Records Addition
URL: https://github.com/apache/kafka/pull/8220
After https://github.com/apache/kafka/pull/7312/, we could still return data during the rebalance phase, which means it could be possible to find records without corresponding tasks. Do not throw illegal state here and just ignore should be safe.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Records could not find corresponding partition/task
> ---------------------------------------------------
>
> Key: KAFKA-9645
> URL: https://issues.apache.org/jira/browse/KAFKA-9645
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.6.0
> Reporter: Boyang Chen
> Assignee: Boyang Chen
> Priority: Major
>
> We could be hitting the illegal state when stream kicks off a rebalance with all tasks closed:
> ```
> [2020-03-03T18:36:09-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,105] WARN [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Detected that the thread is being fenced. This implies that this thread missed a rebalance and dropped out of the consumer group. Will close out all assigned tasks and rejoin the consumer group. (org.apache.kafka.streams.processor.internals.StreamThread)
> [2020-03-03T18:36:09-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,105] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Consumer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-restore-consumer, groupId=null] Subscribed to partition(s): stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000049-changelog-2 (org.apache.kafka.clients.consumer.KafkaConsumer)
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,286] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Producer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-1_1-producer, transactionalId=stream-soak-test-1_1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,287] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] task [1_1] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,287] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Consumer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions (org.apache.kafka.clients.consumer.KafkaConsumer)
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,290] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Producer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-3_2-producer, transactionalId=stream-soak-test-3_2] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,292] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] task [3_2] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,293] ERROR [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Unable to locate active task for received-record partition node-name-repartition-1. Current tasks: TaskManager
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) > MetadataState:
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) > Tasks:
> (org.apache.kafka.streams.processor.internals.StreamThread)
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,293] ERROR [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Encountered the following exception during processing and the thread is going to shut down: (org.apache.kafka.streams.processor.internals.StreamThread)
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) java.lang.NullPointerException: Task was unexpectedly missing for partition node-name-repartition-1
> at org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:984)
> at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:820)
> ```
> We should be more careful in this case by avoiding processing data when no corresponding task is found.
> at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:751)
> at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:725)
> [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,294] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.processor.internals.StreamThread)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)