[jira] [Updated] (STORM-42) Support sending arbitrary data types back and forth through DRPC

["Rick Kellogg (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rick Kellogg updated STORM-42:
------------------------------
    Component/s: storm-core

> Support sending arbitrary data types back and forth through DRPC
> ----------------------------------------------------------------
>
>                 Key: STORM-42
>                 URL: https://issues.apache.org/jira/browse/STORM-42
>             Project: Apache Storm
>          Issue Type: Improvement
>          Components: storm-core
>            Reporter: James Xu
>
> https://github.com/nathanmarz/storm/issues/41
> Right now, you can only send Strings back and forth due to the limitations of Thrift. It would be nice if DRPC were structured in a way so that you could put any interface you want on it. So the Thrift interface would be one implementation, and another implementation could just send Java objects back and forth (perhaps by using Kryo).
> -----------------------------------------------------------------------------------------------------
> thatsnotright: Why would you define the DRPCRequest to have function args as a string? What is the limitation I am not seeing of Thrift? There already exists kryo and java serialization support, why not define func_args to be binary and serialize in to that?
> -----------------------------------------------------------------------------------------------------
> dmk23: So are there any specific plans to make this feature happen? Thrift obviously can be used to define objects of arbitrary complexity and this would also allow for cross-language DRPC calls.
> We currently have a set of custom Thrift services written in Java that are called from Python clients. We'd be interested in converting our Java backend into Storm topologies but RPC data type limitation is a significant barrier, especially given that this was to work across languages.
> -----------------------------------------------------------------------------------------------------
> nathanmarz: Yea, the right solution to this is to just do binary inputs and outputs through DRPC. Actually it should be something like:
> list<list<binary>> execute(string function, binary args)
> Then you can do the serialization/deserialization around that. The return type there is because Trident returns a list of tuples for the output of drpc, where each tuple is a list of values. Each value is in serialized form here.
> -----------------------------------------------------------------------------------------------------
> dmk23: I understand anything can be manually serialized / deserialized and pushed through the existing "execute" call.
> But this looks quite clumsy / redundant since these DRPC service definitions could be easily expressed with Thrift, then converted into easy-to-use generated code in any language and imported into Storm through Java. Given that Thrift is already used within Storm this seems like an obvious solution.
> Would be nice to have support for having Storm import arbitrary Thrfit service definition and have DRPC services natively expose interfaces / data structures as defined in those IDLs...
> -----------------------------------------------------------------------------------------------------
> suchetchachra:A temporary workaround for the string size limitation (64K) could be to save the object to be returned to a distributed cache and return the key (String). On the client side, the key can be used to retrieve the object from the cache.
> -----------------------------------------------------------------------------------------------------
> dimazaur: Hello, Nathan.
> Is there any plans about drpc binary support?
> We are thinking about move to Storm system on our servers.
> And this String type limitation is blocking us.
> Thanks a lot.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)