You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Yichi Zhang (Jira)" <ji...@apache.org> on 2021/03/29 19:04:00 UTC
[jira] [Updated] (BEAM-8998) Avoid excessive bundle progress
polling in Dataflow Runner
[ https://issues.apache.org/jira/browse/BEAM-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yichi Zhang updated BEAM-8998:
------------------------------
Status: Open (was: Triage Needed)
> Avoid excessive bundle progress polling in Dataflow Runner
> ----------------------------------------------------------
>
> Key: BEAM-8998
> URL: https://issues.apache.org/jira/browse/BEAM-8998
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Yichi Zhang
> Priority: P3
>
> Dataflow Java runner uses 0.1 secs interval for polling bundle progress from SDK Harness, and use the result to decide whether data transfer should be throttled. This can potentially overload SDK Harness.
> We should try to come up with a way to avoid the throttling and lower the bundle progress request frequency significantly.
>
> Code reference:
> frequency setting: [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java#L296]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)