[GitHub] samza pull request #552: SAMZA-1745: Remove all usages of StreamSpec and App...

[prateekm <gi...@git.apache.org>]

GitHub user prateekm opened a pull request:

    https://github.com/apache/samza/pull/552

    SAMZA-1745: Remove all usages of StreamSpec and ApplicationRunner from the operator spec and impl layers.

    This PR is a pre-requisite for adding support for user-provided SystemDescriptors and StreamDescriptors to the High Level API.
    
    It removes all usages of StreamSpec and ApplicationRunner from the OperatorSpec and OperatorImpl layers. DAG specification (StreamGraphSpec, OperatorSpecs) now only relies on logical streamIds (and in future, will use the user-provided StreamDescriptors). DAG execution (i.e., StreamOperatorTask, OperatorImpls) now only relies on logical streamIds and their corresponding SystemStreams, which are obtained using StreamConfig in OperatorImplGraph.
    
    After this change, StreamSpec can be thought of as the API between StreamManager and SystemAdmins for creating and validating streams. Ideally ExecutionPlanner shouldn't rely on StreamSpec either, but it currently does so extensively, so I'll leave that refactor for later.
    
    Additional changes:
    1. ApplicationRunner is no longer responsible for creating/returning StreamSpec instances. Instances can be created directly using the StreamSpec constructors, or by using one of the util methods in the new StreamUtil class.
    
    2. StreamSpec class no longer tracks the isBroadcast and isBounded status for streams. 
    The former was being used for communicating broadcast status from the StreamGraphSpec to the planner so that it could write the broadcast input configurations. This is now done using a separate Set of broadcast streamIds in StreamGraphSpec.
    The latter was being set by the ApplicationRunner based on a config, and then passed to the planner so that it could write the bounded input configs. This was redundant, so I removed it.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/prateekm/samza stream-spec-cleanup

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/samza/pull/552.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #552
    
----
commit 8e6fc2dac5af11f0748fdcd32175c2ae95418fc7
Author: prateekm <pr...@...>
Date:   2018-06-09T22:01:10Z

    Remove all usages of StreamSpec and ApplicationRunner from the operator spec and impl layers.

----


---

[GitHub] samza pull request #552: SAMZA-1745: Remove all usages of StreamSpec and App...

[asfgit <gi...@git.apache.org>]

Github user asfgit closed the pull request at:

    https://github.com/apache/samza/pull/552


---