You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Jan Lukavský (JIRA)" <ji...@apache.org> on 2019/07/26 20:45:00 UTC
[jira] [Comment Edited] (BEAM-7785) DirectRunner: watermarks are
updated asynchronously from bundle processing
[ https://issues.apache.org/jira/browse/BEAM-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894116#comment-16894116 ]
Jan Lukavský edited comment on BEAM-7785 at 7/26/19 8:44 PM:
-------------------------------------------------------------
Hmmm ... looking into that - maybe the approach you suggest would not be that complex as I suspected. Because QuiescenceDriver has reference to EvaluationContext, _maybe_ it would be possible to copy WatermarkManager inside the EvaluationContext before bundle and free it in bundle finish callback. Would you find that possible?
was (Author: je-ik):
Hmmm ... looking into that - maybe the approach you suggest would not be that complex as I suspected. Because QuiescenceDriver has reference on EvaluationContext, _maybe_ it would be possible to copy WatermarkManager inside the EvaluationContext before bundle and free it in bundle finish callback. Would you find that possible?
> DirectRunner: watermarks are updated asynchronously from bundle processing
> --------------------------------------------------------------------------
>
> Key: BEAM-7785
> URL: https://issues.apache.org/jira/browse/BEAM-7785
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Affects Versions: 2.13.0
> Reporter: Jan Lukavský
> Assignee: Jan Lukavský
> Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Watermarks are updated in QuiescenceDriver (by calling fireTimers, which calls forceRefresh()) on WatermarkManager. This results in creating timer bundles, that are then processed asynchronously as DirectTransformExecutor. Because of that, watermarks (input watermarks mostly) might be updated while bundle is being processed. That violates assumption, that bundle processing should be atomical (with identical external conditions during processing of whole bundle).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)