[jira] [Created] (FLINK-3514) Add support for slowly changing streaming broadcast variables

["Stephan Ewen (JIRA)" <ji...@apache.org>]

Stephan Ewen created FLINK-3514:
-----------------------------------

             Summary: Add support for slowly changing streaming broadcast variables
                 Key: FLINK-3514
                 URL: https://issues.apache.org/jira/browse/FLINK-3514
             Project: Flink
          Issue Type: New Feature
          Components: Streaming
            Reporter: Stephan Ewen


With "slow changing inputs", I mean auxiliary data sets that change over time. Think about dictionaries, set of rules, etc that are updates at slow rates and then applied onto another stream.

They are a bit like broadcast variables in the DataSet API, but not constant over the entire lifetime of the job.

One can currently approximate this via connected streams:
{code}
DataStream<Rule> rules = ...
DataStream<Event> events = ...

rules.broadcast().connect(events).flatMap(...);
{code}

The disadvantage is that one has no control that the broadcasted "rules" input is materialized in the function before the "events" are delivered.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)