[jira] [Updated] (STORM-561) Add ability to create topologies dynamically

["Nathan Leung (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nathan Leung updated STORM-561:
-------------------------------
    Description: 
It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc).

I would propose the following data structures for building core storm topologies.  I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it):

{code}

// fields value and arguments are mutually exclusive
class Argument {
    String argumentType;  // Class used to lookup arguments in method/constructor
    String implementationType; // Class used to create this argument
    String value; // String used to construct this argument
    List<Argument> arguments; // arguments used to build this argument
}

class Dependency {
    String upstreamComponent; // name of upstream component
    String grouping;
    List<Argument> arguments; // arguments for the grouping
}

class StormSpout {
    String name;
    String klazz;  // Class of this spout
    List <Argument> arguments;
    int numTasks;
    int numExecutors;
}

class StormBolt {
    String name;
    String klazz; // Class of this bolt
    List <Argument> arguments;
    int numTasks;
    int numExecutors;
    List<Dependency> dependencies;
}

class StormTopologyRepresentation {
    String name;
    List<StormSpout> spouts;
    List<StormBolt> bolts;
    Map config;
    int numWorkers;
}

{code}

Topology creation will be built on top of the data structures above.  The benefits:

* Dependency free.  Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation.
* support arbitrary spout and bolts types
* support of all groupings, streams, via reflections
* ability to specify configuration map via config file
* reification of spout / bolt / dependency arguments
** recursive argument reification for complex objects

  was:
It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc).

I would propose the following data structures for building core storm topologies.  I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it):

{code}

// fields value and arguments are mutually exclusive
class Argument {
    Class argumentType;  // type used to lookup arguments in method/constructor
    Class implementationType; // type used to create this argument
    String value; // String used to construct this argument
    List<Argument> arguments; // arguments used to build this argument
}

class Dependency {
    String upstreamComponent; // name of upstream component
    String grouping;
    List<Argument> arguments; // arguments for the grouping
}

class StormSpout {
    String name;
    Class class;  // class of this spout
    int numTasks;
    int numExecutors;
}

class StormBolt {
    String name;
    Class class;
    int numTasks;
    int numExecutors;
    List<Dependency> dependencies;
}

class StormTopologyRepresentation {
    String name;
    List<StormSpout> spouts;
    List<StormBolt> bolts;
    Map config;
    int numWorkers;
}

{code}

Topology creation will be built on top of the data structures above.  The benefits:

* Dependency free.  Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation.
* support arbitrary spout and bolts types
* support of all groupings, streams, via reflections
* ability to specify configuration map via config file
* reification of spout / bolt / dependency arguments
** recursive argument reification for complex objects


> Add ability to create topologies dynamically
> --------------------------------------------
>
>                 Key: STORM-561
>                 URL: https://issues.apache.org/jira/browse/STORM-561
>             Project: Apache Storm
>          Issue Type: Improvement
>            Reporter: Nathan Leung
>            Assignee: Nathan Leung
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc).
> I would propose the following data structures for building core storm topologies.  I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it):
> {code}
> // fields value and arguments are mutually exclusive
> class Argument {
>     String argumentType;  // Class used to lookup arguments in method/constructor
>     String implementationType; // Class used to create this argument
>     String value; // String used to construct this argument
>     List<Argument> arguments; // arguments used to build this argument
> }
> class Dependency {
>     String upstreamComponent; // name of upstream component
>     String grouping;
>     List<Argument> arguments; // arguments for the grouping
> }
> class StormSpout {
>     String name;
>     String klazz;  // Class of this spout
>     List <Argument> arguments;
>     int numTasks;
>     int numExecutors;
> }
> class StormBolt {
>     String name;
>     String klazz; // Class of this bolt
>     List <Argument> arguments;
>     int numTasks;
>     int numExecutors;
>     List<Dependency> dependencies;
> }
> class StormTopologyRepresentation {
>     String name;
>     List<StormSpout> spouts;
>     List<StormBolt> bolts;
>     Map config;
>     int numWorkers;
> }
> {code}
> Topology creation will be built on top of the data structures above.  The benefits:
> * Dependency free.  Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation.
> * support arbitrary spout and bolts types
> * support of all groupings, streams, via reflections
> * ability to specify configuration map via config file
> * reification of spout / bolt / dependency arguments
> ** recursive argument reification for complex objects



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)