You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Thomas Groh (JIRA)" <ji...@apache.org> on 2017/06/14 17:22:00 UTC
[jira] [Resolved] (BEAM-2398) Increasing latency within
DirectRunner caused by cumulated TransformWatermarks
[ https://issues.apache.org/jira/browse/BEAM-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Groh resolved BEAM-2398.
-------------------------------
Resolution: Fixed
Fix Version/s: 2.1.0
> Increasing latency within DirectRunner caused by cumulated TransformWatermarks
> ------------------------------------------------------------------------------
>
> Key: BEAM-2398
> URL: https://issues.apache.org/jira/browse/BEAM-2398
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Affects Versions: 2.0.0
> Reporter: Antony Mayi
> Assignee: Thomas Groh
> Fix For: 2.1.0
>
> Attachments: LatencyTest.java
>
>
> Over the time the end-to-end latency of a pipeline running on DirectRunner is significantly increasing.
> This is caused by ever growing sets of:
> * {{WatermarkManager.TransformWatermarks.inputWatermark.pendingElements}}
> * {{WatermarkManager.TransformWatermarks.synchronizedProcessingInputWatermark.pendingBundles}}
> That means calls to {{WatermarkManager.TransformWatermarks.refresh()}} which need to iterate through that collections take longer and longer and the latency is growing.
> I believe it is the line {{WaterMark.updatePending()}} line:
> {quote}
> if (input != null) {
> // Add the unprocessed inputs
> completedTransform.addPending(result.getUnprocessedInputs());
> {quote}
> that's adding the items that are never removed.
> See attached [^LatencyTest.java] demo code showing the increasing latency.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)