You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "David Arthur (Jira)" <ji...@apache.org> on 2024/09/13 22:59:00 UTC
[jira] [Comment Edited] (KAFKA-17553) DefaultTaskManager.shutdown stalling on Github CI
[ https://issues.apache.org/jira/browse/KAFKA-17553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881712#comment-17881712 ]
David Arthur edited comment on KAFKA-17553 at 9/13/24 10:58 PM:
----------------------------------------------------------------
Actually this looks like the culprit
{code:java}
"clientId-StreamThread-1-TaskExecutor-1" #12953 prio=5 os_prio=0 cpu=0.59ms elapsed=5651.50s tid=0x00007f44fe5f5f80 nid=0x7d8b2 waiting on condition [0x00007f44967fa000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.12/Native Method)
- parking to wait for <0x00000000e31a06b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@17.0.12/LockSupport.java:341)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.12/AbstractQueuedSynchronizer.java:506)
at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.12/ForkJoinPool.java:3465)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.12/ForkJoinPool.java:3436)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.12/AbstractQueuedSynchronizer.java:1625)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.lambda$awaitProcessableTasks$1(DefaultTaskManager.java:142)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager$$Lambda$5698/0x00007f449cd66d38.get(Unknown Source)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.returnWithTasksLocked(DefaultTaskManager.java:353)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.awaitProcessableTasks(DefaultTaskManager.java:129)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskExecutor$TaskExecutorThread.runOnce(DefaultTaskExecutor.java:119)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskExecutor$TaskExecutorThread.run(DefaultTaskExecutor.java:62) {code}
In DefaultTaskManager#awaitProcessableTasks we are awaiting a condition variable indefinitely.
cc [~mjsax] [~lucasbru]
was (Author: davidarthur):
Actually this looks like the culprit
{code:java}
"clientId-StreamThread-1-TaskExecutor-1" #12953 prio=5 os_prio=0 cpu=0.59ms elapsed=5651.50s tid=0x00007f44fe5f5f80 nid=0x7d8b2 waiting on condition [0x00007f44967fa000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.12/Native Method)
- parking to wait for <0x00000000e31a06b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@17.0.12/LockSupport.java:341)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.12/AbstractQueuedSynchronizer.java:506)
at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.12/ForkJoinPool.java:3465)
at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.12/ForkJoinPool.java:3436)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.12/AbstractQueuedSynchronizer.java:1625)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.lambda$awaitProcessableTasks$1(DefaultTaskManager.java:142)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager$$Lambda$5698/0x00007f449cd66d38.get(Unknown Source)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.returnWithTasksLocked(DefaultTaskManager.java:353)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.awaitProcessableTasks(DefaultTaskManager.java:129)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskExecutor$TaskExecutorThread.runOnce(DefaultTaskExecutor.java:119)
at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskExecutor$TaskExecutorThread.run(DefaultTaskExecutor.java:62) {code}
In DefaultTaskManager#awaitProcessableTasks we are awaiting a condition variable indefinitely.
> DefaultTaskManager.shutdown stalling on Github CI
> -------------------------------------------------
>
> Key: KAFKA-17553
> URL: https://issues.apache.org/jira/browse/KAFKA-17553
> Project: Kafka
> Issue Type: Bug
> Components: build, streams
> Reporter: David Arthur
> Priority: Major
> Attachments: GradleWorkerMain-142716.txt
>
>
> After merging [https://github.com/apache/kafka/pull/17181,] we now have thread dumps available for build timeouts. Coincidentally, the trunk build of that very PR encountered a timeout.
> [https://github.com/apache/kafka/actions/runs/10851711578]
> It looks like we are getting stuck in a shutdown method
> {code:java}
> "stream-thread-test-87bf53a8-54f2-485f-a4b6-acdbec0a8b3d-StreamThread-1" #1 prio=5 os_prio=0 cpu=42316.77ms elapsed=8450.21s tid=0x00007f44fc02c700 nid=0x22d7f in Object.wait() [0x00007f45041fa000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(java.base@17.0.12/Native Method)
> - waiting on <no object reference available>
> at java.lang.Thread.join(java.base@17.0.12/Thread.java:1307)
> - locked <0x00000000e31a0aa8> (a org.apache.kafka.streams.processor.internals.tasks.DefaultTaskExecutor$TaskExecutorThread)
> at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskExecutor.awaitShutdown(DefaultTaskExecutor.java:264)
> at org.apache.kafka.streams.processor.internals.tasks.DefaultTaskManager.shutdown(DefaultTaskManager.java:377) {code}
>
> Attached is the full thread dump
> [^GradleWorkerMain-142716.txt]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)