You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Arun Pandian (Jira)" <ji...@apache.org> on 2022/03/24 17:56:00 UTC

[jira] [Commented] (BEAM-14157) Don't send requests on a closed windmill Grpc streams

    [ https://issues.apache.org/jira/browse/BEAM-14157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17511997#comment-17511997 ] 

Arun Pandian commented on BEAM-14157:
-------------------------------------

Updated with more details on cause and effect.

> Don't send requests on a closed windmill Grpc streams
> -----------------------------------------------------
>
>                 Key: BEAM-14157
>                 URL: https://issues.apache.org/jira/browse/BEAM-14157
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow
>            Reporter: Arun Pandian
>            Priority: P2
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> GrpcWindmillServer could send requests requests on client closed streams. This leads to windmill streams getting stalling occasionally for few seconds to few minutes. grpc-java doc says not to call onNext to send after a stream is client closed. 
> When the streams get stalled it is logged as "Output channel stalled for {}s, outbound thread {}."  from [here|https://github.com/apache/beam/blob/7727dc99ed5dc1fb46166ef496ab3607ee2779f8/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/DirectStreamObserver.java#L100]
> Ref:
> [https://github.com/grpc/grpc-java/blob/master/stub/src/main/java/io/grpc/stub/StreamObserver.java#L62]
> [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java#L939]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)