You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Pejman Hosseini <pe...@gmail.com> on 2015/06/23 13:44:16 UTC
RDMA adn apache storm
Hello everybody!
I want to implement RDMA on Storm as a part of my Thesis inspired by
"Accelerating
Big Data Processing with Hadoop, Spark, and Memcached on Datacenters with
Modern Architectures
<http://www.cse.ohio-state.edu/%7Epanda/isca15_bigdata.pdf>".
Unfortunately I can't find any papers or references that explain or
implemente it. I wanted to know whether it is possible at all?
--
*Seyyed Pejman Hosseini <pe...@gmail.com>*
Re: RDMA adn apache storm
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
It is totally possible. If I were you I would look at the messaging layer that we have. https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/messaging/IContext.java is the interface to implement. The port really just says which process on the remote host to send data to, so you could abuse it how you want. That is the most obvious first step. With the much higher bandwidth of typical RDMA setups we will probably need to spend a decent amount of time optimizing the rest of the worker process to be able to keep up with it. we have plans to do some of that already with STORM-889, but any help with that is appreciated.
- Bobby
On Tuesday, June 23, 2015 6:44 AM, Pejman Hosseini <pe...@gmail.com> wrote:
Hello everybody!
I want to implement RDMA on Storm as a part of my Thesis inspired by
"Accelerating
Big Data Processing with Hadoop, Spark, and Memcached on Datacenters with
Modern Architectures
<http://www.cse.ohio-state.edu/%7Epanda/isca15_bigdata.pdf>".
Unfortunately I can't find any papers or references that explain or
implemente it. I wanted to know whether it is possible at all?
--
*Seyyed Pejman Hosseini <pe...@gmail.com>*