You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Mark Shields (JIRA)" <ji...@apache.org> on 2016/05/12 19:09:12 UTC
[jira] [Commented] (BEAM-175) Leak garbage collection timers in
GlobalWindow
[ https://issues.apache.org/jira/browse/BEAM-175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281929#comment-15281929 ]
Mark Shields commented on BEAM-175:
-----------------------------------
Here's the proposed solution, which pull/139 prototypes:
(Note that pull/139 also fixes https://issues.apache.org/jira/browse/BEAM-32 and https://issues.apache.org/jira/browse/BEAM-122 since they are all in the same space.)
- Window.ClosingBehavior becomes Window.EmptyPaneBehavior
- New property WindowingStrategy.onTimeBehavior of type Window.EmptyPaneBehavior, which is the analogue of closingBehavior but for the ON_TIME pane. Default is FIRE_ALWAYS.
- New enum Window.PaneIndexBehavior { SEQUENTIAL, ZERO }
- New property WindowingStrategy.paneIndexBehavior of type Window.PaneIndexBehavior which controls how PaneInfo.getIndex and friends are determined. Default is SEQUENTIAL.
- In ReduceFnRunner.emitIfApproriate we delete any outstanding end-of-window or garbage-collection timer if we can be sure the corresponding onTimer would be a no-op. This means:
- No element state, either combiner or iterable.
- No end-of-window/garbage-collection holds, which will have been elided if the onTimeBehavior/closingBehavor is FIRE_IF_NON_EMPTY
- No trigger finished bits (ie the trigger must be a Repeatably which never closes).
- No previous pane info, which will have been elided if paneIndexBehavior is ZERO.
- No active window state, ie the windows are not merging or there's no currently active merging window.
The upshot is the following classic pattern will never leave behind state per key:
Window.<KV<Long, Void>>into(new GlobalWindows())
.triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(10))))
.discardingFiredPanes()
.onTimeBehavior(EmptyPaneBehavior.FIRE_IF_NON_EMPTY)
.paneIndexBehavior(PaneIndexBehavior.ZERO)
.withAllowedLateness(Duration.ZERO))
> Leak garbage collection timers in GlobalWindow
> ----------------------------------------------
>
> Key: BEAM-175
> URL: https://issues.apache.org/jira/browse/BEAM-175
> Project: Beam
> Issue Type: Bug
> Components: runner-core
> Reporter: Mark Shields
> Assignee: Mark Shields
>
> Consider the transform:
> Window
> .into(new GlobalWindows())
> .triggering(
> Repeatedly.forever(
> AfterProcessingTime.pastFirstElementInPane().plusDelayOf(...)))
> .discardingFiredPanes()
> This is a common idiom for 'process elements bunched by arrival time'.
> Currently we create an end-of-window timer per key, which clearly will only fire if the pipeline is drained.
> Better would be to avoid creating end-of-window timers if there's no state which needs to be processed at end-of-window (ie at drain if the Global window).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)