[GitHub] nwangtw commented on a change in pull request #3029: Nwang/add bolt based flatmap streamlet

[GitBox <gi...@apache.org>]

nwangtw commented on a change in pull request #3029: Nwang/add bolt based flatmap streamlet
URL: https://github.com/apache/incubator-heron/pull/3029#discussion_r218888775
 
 

 ##########
 File path: heron/api/src/java/org/apache/heron/streamlet/Streamlet.java
 ##########
 @@ -85,6 +86,13 @@
   <T> Streamlet<T> flatMap(
       SerializableFunction<R, ? extends Iterable<? extends T>> flatMapFn);
 
+  /**
+   * Return a new Streamlet by applying a bolt to each element of this Streamlet and
+   * flattening the result
+   * @param flatMapBolt The rich bolt that is used to emit tuples from each input element
+   */
+  <T> Streamlet<T> flatMap(IRichBolt flatMapBolt);
 
 Review comment:
   First of all, discussion is welcome in this PR (ok to move to mailing list as well). My thoughts are not firm yet but I do believe we need it.
   
   Here are my thoughts:
   
   Streamlet is not really the abstraction. My feeling is that Streamlet is good at the DAG layer but not flexible enough in the low level (operators). I would think it is like Scala vs Java(not the same, just some idea). Scala has the nice functional API but it is pretty useless in real life if procedural code is not allowed/supported.
   
   Two reasons:
   1. Migration is one major reason. There are quite some existing topologies written in low level API (for heron and storm). Streamlet is only friendly to new users, existing code such as KafkaSpout (it is spout, but same issue) in storm and some ML bolts has to be rewritten to take the readability/maintainability advantages.
   2. Bolt/Spout are more flexible. They can do a lot more than a function provided by Streamlet API (initialization, config, checkpoint, etc). For examples, the stateful processing and component configs, they are not supported currently by Streamlet and if we add the features, it is likely user has to provide extra functions as parameters and the Streamlet API would became more and more complicated. Streamlet API will evolve but supporting Bolt/Spout could give us a lot room to design a clean API.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services