[jira] [Resolved] (BEAM-1721) Reshuffle can shift elements in time

["Thomas Groh (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Groh resolved BEAM-1721.
-------------------------------
       Resolution: Fixed
    Fix Version/s: First stable release

> Reshuffle can shift elements in time
> ------------------------------------
>
>                 Key: BEAM-1721
>                 URL: https://issues.apache.org/jira/browse/BEAM-1721
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Thomas Groh
>            Assignee: Thomas Groh
>             Fix For: First stable release
>
>
> The reshuffle transform is meant to have no visible effects on the data that it processes. However, due to the use of a {{GroupByKey}}, the timestamp of the output elements is determined by the {{OutputTimeFn}} of the input {{WindowingStrategy}}
> Elements should not be shifted in time when being processed in {{Reshuffle}}. Currently this would require reifying all timestamps before applying the GroupByKey and reapplying them after. As an intermediate solution, elements should never be shifted forwards in time, as doing so permits the watermark to advance improperly (if the elements already contain their timestamps, for example), and prevents the timestamps from being reassigned within a {{DoFn}} or via the {{WithTimestamps}} transform.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)