[jira] [Commented] (STORM-151) Support multilang in Trident

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14214148#comment-14214148 ] 

Jungtaek Lim commented on STORM-151:
------------------------------------

I'm interested to this issue. Is it still valid?

Btw, I read description of this issue, but I don't understand what Nathan said.
(Currently I can only understand colinsurprenant's first approach, and it could be valid if we don't think about performance.)

Could you share some sketches if you don't mind?
Thanks in advance!

> Support multilang in Trident
> ----------------------------
>
>                 Key: STORM-151
>                 URL: https://issues.apache.org/jira/browse/STORM-151
>             Project: Apache Storm
>          Issue Type: Improvement
>            Reporter: James Xu
>
> https://github.com/nathanmarz/storm/issues/353
> Since it's impractical to have a separate subprocess for every operation packed into a bolt, I think the best tradeoff is to allow multilang operations that will just be their own bolt. This is a low level exposure of the API and the multilang program will have to make sure to emit the batch id as the first field. Multilang operations would be treated similar to aggregators in that the tuples they produce are brand new with brand new fields.
> ----------
> colinsurprenant: I am trying to wrap my head around this issue. Would it make sense to expose classes like ShellFunction, ShellAggregator, ShellFilter, etc?
> A topology definition could be something like:
> topology.newDRPCStream("words")
>       .each(new Fields("args"), new ShellFunction("ruby", "split.rb"), new Fields("word"))
>       .groupBy(new Fields("word"))
>       .stateQuery(wordCounts, new Fields("word"), new MapGet(), new Fields("count"))
>       .each(new Fields("count"), new ShellFilter("ruby", "somefilter.rb"))
>       .aggregate(new Fields("count"), new ShellCombinerAggregator("ruby", "someaggregator.rb"), new Fields("sum"));
> Then the builder would assign a new shell bolt for each of these shell operations.
> ----------
> nathanmarz: That would be one way to go about it – though that will have a lot of overhead with serializing back and forth between Java-land and multilang-land. This is especially true if you have many operations running in the same bolt (like 5 eaches in a row). Another way to go about it would be to expose a lower level facility, where the shell process is a regular bolt and has full control over the input and output tuples. Whatever fields it emits will be the fields available in the output tuples. Additionally, one of Trident's constraints is that the first field of output tuples contain the "batch id".
> ----------
> colinsurprenant:Right. My proposition is basically what you described as "impractical to have a separate subprocess for every operation" in the issue description.
> So what you are suggesting is to just expose a ShellBolt, which would basically respect the same semantic as the general Aggregator operation in that it can emit any number of tuples with any number of fields in regard to to input batch.
> ----------
> joshbronson: I'm probably missing something. But would exposing an interface for an aggregator and using it carefully to avoid spawning too many processes accomplish something similar, without having to worry about batch ids?
> We've run up against this problem recently and come up with a couple of ways to get around the many-processes issue. We've considered spawning a server that would listen on a standard TCP port range or Unix sockets. Is there a place to hook in, other than the prepare method, to launch such a beast? Other functions would presumably use their prepare method to discover and connect to the server, and it would be nice to avoid races or locks.
> Also, we're working on an adapter for "BaseFunction" at the moment, though it's not quite ready for public use, and I wonder if anything will go wrong if a function continues to emit tuples to its collector after its execute method has returned. We're operating under the assumption that something will, so we've required the shell command the tell me when it's done responding to the input we just gave it. Currently we do that by requiring it to emit a JSON array. If we go with the server approach, we may also require the script to handle a tab-delimited ID telling the server to delegate to the appropriate handler for a function. Presumably this would require a thin library in languages used to script storm. There seems to be an appetite here for open-sourcing the components required to do this in Ruby; we definitely want to make this easy for folks.
> ----------
> quintona: Hi Guys, I am needing the multilang support. From what I gather, the idea would be to hook the shell bolt into the trident topology, and have it ignore and handle the batch id field?
> If that understanding is correct, then it raises the interesting question around support for any bolt within a Trident topology.
> ----------
> nathanmarz: Well, the right way to go is to essentially expose the Aggregator interface via multilang. This is general enough to encompass functions, filters, and aggregation. We could also allow users to mark multilang aggregators as committers so that they can achieve exactly-once semantics when interacting with external state.
> ----------
> quintona: I wasn't intending to allow the updating of external state via a multilang function. Surely that is a multilang state discussion, which is fundamentally different? Or am I missing something?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)