[jira] [Resolved] (STORM-561) Add ability to create topologies dynamically

["P. Taylor Goetz (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

P. Taylor Goetz resolved STORM-561.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: 0.10.0

> Add ability to create topologies dynamically
> --------------------------------------------
>
>                 Key: STORM-561
>                 URL: https://issues.apache.org/jira/browse/STORM-561
>             Project: Apache Storm
>          Issue Type: Improvement
>            Reporter: Nathan Leung
>            Assignee: P. Taylor Goetz
>             Fix For: 0.10.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc).
> I would propose the following data structures for building core storm topologies.  I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it):
> {code}
> // fields value and arguments are mutually exclusive
> class Argument {
>     String argumentType;  // Class used to lookup arguments in method/constructor
>     String implementationType; // Class used to create this argument
>     String value; // String used to construct this argument
>     List<Argument> arguments; // arguments used to build this argument
> }
> class Dependency {
>     String upstreamComponent; // name of upstream component
>     String grouping;
>     List<Argument> arguments; // arguments for the grouping
> }
> class StormSpout {
>     String name;
>     String klazz;  // Class of this spout
>     List <Argument> arguments;
>     int numTasks;
>     int numExecutors;
> }
> class StormBolt {
>     String name;
>     String klazz; // Class of this bolt
>     List <Argument> arguments;
>     int numTasks;
>     int numExecutors;
>     List<Dependency> dependencies;
> }
> class StormTopologyRepresentation {
>     String name;
>     List<StormSpout> spouts;
>     List<StormBolt> bolts;
>     Map config;
>     int numWorkers;
> }
> {code}
> Topology creation will be built on top of the data structures above.  The benefits:
> * Dependency free.  Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation.
> * support arbitrary spout and bolts types
> * support of all groupings, streams, via reflections
> * ability to specify configuration map via config file
> * reification of spout / bolt / dependency arguments
> ** recursive argument reification for complex objects



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)