[jira] [Commented] (STORM-561) Add ability to create topologies dynamically

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566854#comment-14566854 ] 

ASF GitHub Bot commented on STORM-561:
--------------------------------------

Github user HeartSaVioR commented on a diff in the pull request:

    https://github.com/apache/storm/pull/546#discussion_r31396529
  
    --- Diff: external/flux/flux-examples/README.md ---
    @@ -0,0 +1,68 @@
    +# Flux Examples
    +A collection of examples illustrating various capabilities.
    +
    +## Building From Source and Running
    +
    +Checkout the projects source and perform a top level Maven build (i.e. from the `flux` directory):
    +
    +```bash
    +git clone https://github.com/apache/storm.git
    +cd storm
    +mvn install -DskipTests=true
    +```
    +
    +This will create a shaded (i.e. "fat" or "uber") jar in the `external/flux/flux-examples/target` directory that can run/deployed with
    +the `storm` command:
    +
    +```bash
    +cd flux-examples
    +storm jar ./target/flux-examples-*-SNAPSHOT.jar org.apache.storm.flux.Flux --local ./src/main/resources/simple_wordcount.yaml
    +```
    +
    +The example YAML files are also packaged in the examples jar, so they can also be referenced with Flux's `--resource`
    +command line switch:
    +
    +```bash
    +storm jar ./target/flux-examples-*.jar org.apache.storm.flux.Flux --local --resource /simple_wordcount.yaml
    +```
    +
    +## Available Examples
    +
    +### [simple_wordcount.yaml](src/main/resources/simple_wordcount.yaml)
    +
    +This is a very basic wordcount example using Java spouts and bolts. It simply logs the running count of each word
    +received.
    +
    +### [multilang.yaml](src/main/resources/multilang.yaml)
    +
    +Another wordcount example that uses a spout written in JavaScript (node.js), a bolt written in Python, and two bolts
    +written in java.
    +
    +### [kafka_spout.yaml](src/main/resources/kafka_spout.yaml)
    +This example illustrates how to configure Storm's `storm-kafka` spout using Flux YAML DSL `components`, `references`,
    +and `constructor arguments` constructs.
    +
    +### [simple_hdfs.yaml](src/main/resources/simple_hdfs.yaml)
    +
    +This example demonstrates using Flux to setup a storm-hdfs bolt to write to an HDFS cluster. It also demonstrates Flux's
    +variable substitution/filtering feature.
    +
    +To run the `simple_hdfs.yaml` example, copy the `hdfs_bolt.properties` file to a convenient location and change, at
    +least, the property `hdfs.url` to point to a HDFS cluster. Then you can run the example something like:
    +
    +```bash
    +storm jar ./target/flux-examples-*.jar org.apache.storm.flux.Flux --local ./src/main/resources/simple_hdfs.yaml --filter my_hdfs_bolt.properties
    +```
    +
    +### [simple_hbase.yaml](src/main/resources/simple_hbase.yaml)
    +
    +This example illustrates how to use Flux to setup a storm-hbase bolt to write to HBase.
    +
    +In order to use this example, you will need to edit the `src/main resrouces/hbase-site.xml` file to reflect your HBase
    --- End diff --
    
    @ptgoetz Just a reminder, you may want to check it since it contains typo (resrouces) and hbase-site.xml file is not on directory. And actually we can run simple_hbase.yaml with similar steps from simple_hdfs.yaml.


> Add ability to create topologies dynamically
> --------------------------------------------
>
>                 Key: STORM-561
>                 URL: https://issues.apache.org/jira/browse/STORM-561
>             Project: Apache Storm
>          Issue Type: Improvement
>            Reporter: Nathan Leung
>            Assignee: P. Taylor Goetz
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc).
> I would propose the following data structures for building core storm topologies.  I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it):
> {code}
> // fields value and arguments are mutually exclusive
> class Argument {
>     String argumentType;  // Class used to lookup arguments in method/constructor
>     String implementationType; // Class used to create this argument
>     String value; // String used to construct this argument
>     List<Argument> arguments; // arguments used to build this argument
> }
> class Dependency {
>     String upstreamComponent; // name of upstream component
>     String grouping;
>     List<Argument> arguments; // arguments for the grouping
> }
> class StormSpout {
>     String name;
>     String klazz;  // Class of this spout
>     List <Argument> arguments;
>     int numTasks;
>     int numExecutors;
> }
> class StormBolt {
>     String name;
>     String klazz; // Class of this bolt
>     List <Argument> arguments;
>     int numTasks;
>     int numExecutors;
>     List<Dependency> dependencies;
> }
> class StormTopologyRepresentation {
>     String name;
>     List<StormSpout> spouts;
>     List<StormBolt> bolts;
>     Map config;
>     int numWorkers;
> }
> {code}
> Topology creation will be built on top of the data structures above.  The benefits:
> * Dependency free.  Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation.
> * support arbitrary spout and bolts types
> * support of all groupings, streams, via reflections
> * ability to specify configuration map via config file
> * reification of spout / bolt / dependency arguments
> ** recursive argument reification for complex objects



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)