[jira] [Commented] (STORM-561) Add ability to create topologies dynamically

["P. Taylor Goetz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14366125#comment-14366125 ] 

P. Taylor Goetz commented on STORM-561:
---------------------------------------

FWIW, based on feedback from a number of sources (including comments above), I implemented a POC that is available here:

https://github.com/ptgoetz/flux

The README should provide a pretty good explanation of everything it supports.

It uses YAML for topology definitions, but since JSON is a subset of YAML, that could easily be supported as well. 

One of the goals was to allow developers to use existing spouts and bolts without have to write custom wrappers, etc. For example, you can use the stock kafka spout without writing any custom code. Dealing with components that use builder/configuration methods is not yet implemented, but has been noted in the github issues.

Also, the intent was that this could be added as a component in /external. It is not intended to be part of the core storm codebase. Though there are some ideas present that might be beneficial if included as part of Storm (like the shell spout/bolt extensions).

Any feedback is welcome.

> Add ability to create topologies dynamically
> --------------------------------------------
>
>                 Key: STORM-561
>                 URL: https://issues.apache.org/jira/browse/STORM-561
>             Project: Apache Storm
>          Issue Type: Improvement
>            Reporter: Nathan Leung
>            Assignee: Nathan Leung
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc).
> I would propose the following data structures for building core storm topologies.  I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it):
> {code}
> // fields value and arguments are mutually exclusive
> class Argument {
>     String argumentType;  // Class used to lookup arguments in method/constructor
>     String implementationType; // Class used to create this argument
>     String value; // String used to construct this argument
>     List<Argument> arguments; // arguments used to build this argument
> }
> class Dependency {
>     String upstreamComponent; // name of upstream component
>     String grouping;
>     List<Argument> arguments; // arguments for the grouping
> }
> class StormSpout {
>     String name;
>     String klazz;  // Class of this spout
>     List <Argument> arguments;
>     int numTasks;
>     int numExecutors;
> }
> class StormBolt {
>     String name;
>     String klazz; // Class of this bolt
>     List <Argument> arguments;
>     int numTasks;
>     int numExecutors;
>     List<Dependency> dependencies;
> }
> class StormTopologyRepresentation {
>     String name;
>     List<StormSpout> spouts;
>     List<StormBolt> bolts;
>     Map config;
>     int numWorkers;
> }
> {code}
> Topology creation will be built on top of the data structures above.  The benefits:
> * Dependency free.  Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation.
> * support arbitrary spout and bolts types
> * support of all groupings, streams, via reflections
> * ability to specify configuration map via config file
> * reification of spout / bolt / dependency arguments
> ** recursive argument reification for complex objects



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)