[jira] [Commented] (ATLAS-182) Add data model for Storm topology elements

["Aaron Dossett (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ATLAS-182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960977#comment-14960977 ] 

Aaron Dossett commented on ATLAS-182:
-------------------------------------

Comments about ATLAS-182.patch:

- The code refers to the Storm DAG.  Storm topologies can contain cycles,
so it¹s not literally a DAG.  I don¹t think this has any implications for
the code I see though.

- The type hierarchy looks like NODE -> SPOUT -> BOLT.  I think it would
be better to have SPOUT and BOLT both be subtypes of NODE since SPOUT has
some behaviors and characteristics that a BOLT does not.

- Each NODE also has a set of configurations options associated with it.
These could be primitives (e.g. ³debug=true² or
³namenode=hdfs://nn.target.com²) or more complex configurations such as a
java class describing a file rotation policy for a bolt that writes data
to a file.  I don¹t know if that level of detail is appropriate yet, but
those options are the key metadata about the NODE to reason about its
behavior.

- Some BOLTS act like ³sinks² (although Storm doesn¹t use that term) in
that the output is outside of the storm topology (e.g. writing to HDFS,
streaming to a Hive table, or enqueuing to Kafka). I don¹t know how that
should be reflected in the model. It could be a NODE with no ³outputs² but
whose configuration options would describe where the data goes, or SINK
could be a datatype in the model.  After typing that out, the first option
sounds better.

- I don¹t believe KAFKA and KAFKA TOPIC need to be first class citizens in
this data model.  A Kafka Spout would be represented as a SPOUT with a
JAVA_CLASS of KafkaSpout and a set of configuration options identifying
brokers, topics, etc.

- Very minor, but the patch places the .java files in addons/hive-bridge/,
I assume that should be addons/storm-bridge/?


> Add data model for Storm topology elements
> ------------------------------------------
>
>                 Key: ATLAS-182
>                 URL: https://issues.apache.org/jira/browse/ATLAS-182
>             Project: Atlas
>          Issue Type: Sub-task
>    Affects Versions: 0.6-incubating
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>             Fix For: 0.6-incubating
>
>         Attachments: ATLAS-182.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)