You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Amit Sela (JIRA)" <ji...@apache.org> on 2016/08/10 18:03:20 UTC
[jira] [Closed] (BEAM-542) Spark batch interval should be a
configuration instead of an interpretation of the Pipeline's windows
[ https://issues.apache.org/jira/browse/BEAM-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amit Sela closed BEAM-542.
--------------------------
Resolution: Fixed
Fix Version/s: 0.3.0-incubating
> Spark batch interval should be a configuration instead of an interpretation of the Pipeline's windows
> -----------------------------------------------------------------------------------------------------
>
> Key: BEAM-542
> URL: https://issues.apache.org/jira/browse/BEAM-542
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Reporter: Amit Sela
> Assignee: Amit Sela
> Fix For: 0.3.0-incubating
>
>
> Currently, the SparkRunner extracts the batch interval from the duration of the first window.
> This is wrong in several ways:
> # GlobalWindow pipelines
> # It's an engine specific property and should not be expressed as a part of the logic but rather as a configuration for execution of the pipeline.
> # Effectively forces the definition of Fixed/SlidingWindows even when they are not needed (stateless processing), which also makes the pipeline code not portable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)