[jira] [Created] (FLINK-1618) Add parallel time discretisation for time-window transformations

["Gyula Fora (JIRA)" <ji...@apache.org>]

Gyula Fora created FLINK-1618:
---------------------------------

             Summary: Add parallel time discretisation for time-window transformations 
                 Key: FLINK-1618
                 URL: https://issues.apache.org/jira/browse/FLINK-1618
             Project: Flink
          Issue Type: Improvement
          Components: Streaming
            Reporter: Gyula Fora


Currently discretizers for all windowing policies including time are executed with parallelism 1 when they define global windows. (for instance: sum of the last 10 minutes) 

While this is necessary for arbitrary policies like delta based or user-defined policies. Some discretizers such as Time can be implemented in a distributed fashion.

Distributed time discretisers (and other types) can be implemented in the following way:

-The discretisers should create StreamWindow s with incrementally increasing ID-s starting from the same value so that it is possible to merge them after the transformation
- The partitioner for each discretizer should send the number of partitions created to the merger (the merger should be aware of the number of partitioners present to wait for all the information)
- Based on all the partitioning info the merger can merge the windows properly afterwards





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)