You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2021/05/10 17:20:01 UTC
[jira] [Commented] (BEAM-12144) Dataflow streaming worker stuck and
unable to get work from Streaming Engine
[ https://issues.apache.org/jira/browse/BEAM-12144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342026#comment-17342026 ]
Beam JIRA Bot commented on BEAM-12144:
--------------------------------------
This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned.
> Dataflow streaming worker stuck and unable to get work from Streaming Engine
> ----------------------------------------------------------------------------
>
> Key: BEAM-12144
> URL: https://issues.apache.org/jira/browse/BEAM-12144
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.26.0
> Reporter: Sam Whittle
> Assignee: Sam Whittle
> Priority: P2
> Labels: stale-assigned
>
> Observed in 2.26 but seems like it could affect later versions as well, as previous issues addressing similar problems were before 2.26. This seems similar to BEAM-9651 but not the deadlock observed there.
> The thread getting work has the following stack:
> --- Threads (1): [Thread[DispatchThread,1,main]] State: WAITING stack: ---
> java.base@11.0.9/jdk.internal.misc.Unsafe.park(Native Method)
> java.base@11.0.9/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
> java.base@11.0.9/java.util.concurrent.Phaser$QNode.block(Phaser.java:1127)
> java.base@11.0.9/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
> java.base@11.0.9/java.util.concurrent.Phaser.internalAwaitAdvance(Phaser.java:1057)
> java.base@11.0.9/java.util.concurrent.Phaser.awaitAdvanceInterruptibly(Phaser.java:747)
> app//org.apache.beam.runners.dataflow.worker.windmill.DirectStreamObserver.onNext(DirectStreamObserver.java:49)
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$AbstractWindmillStream.send(GrpcWindmillServer.java:662)
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$GrpcGetWorkStream.onNewStream(GrpcWindmillServer.java:868)
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$AbstractWindmillStream.startStream(GrpcWindmillServer.java:677)
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$GrpcGetWorkStream.(GrpcWindmillServer.java:860)
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer$GrpcGetWorkStream.(GrpcWindmillServer.java:843)
> app//org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.getWorkStream(GrpcWindmillServer.java:543)
> app//org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.streamingDispatchLoop(StreamingDataflowWorker.java:1047)
> app//org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$1.run(StreamingDataflowWorker.java:670)
> java.base@11.0.9/java.lang.Thread.run(Thread.java:834)
> The status page shows:
> GetWorkStream: 0 buffers, 400 inflight messages allowed, 67108864 inflight bytes allowed, current stream is 61355396ms old, last send 61355396ms, last response -1ms
> Showing that the stream was created 17 hours ago, sent the header message but never received a response. With the stack trace it appears that the header was never sent but the stream also didn't terminate with a deadline exceed. This seems like a grpc issue to not get an error for the stream, however it would be safer to not block indefinitely on the Phaser waiting for the send and instead throw an exception after 2x the stream deadline for example.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)