What are Qpid's plans for RDMA?

[gregory james marsh <ma...@cse.ohio-state.edu>]

Hello,

I'm with the Network-Based Computing Lab at Ohio State University.  Our
lab primarily does research with the Message Passing Interface (MPI) over
interconnects like InfiniBand and RDMA.  Recently we have started
evaluating/testing Qpid with the TCP adaptations to InfiniBand (Sockets
Direct Protocol [SDP] and IP over InfiniBand [IPoIB]).

We would like to know more about where the Qpid project plans to go with
RDMA.  Could anyone give me a brief overview and anticipated timeline of
the planned functionality?  I've noticed some RDMA-related code submissions to the C++ trunk
this month by Andrew Stitcher.  I've been able to run the C++ broker
listening for RDMA via the RdmaIOPlugin class (at least according to the
broker log).  However, how does the client side work?  Does the
RdmaConnector class integrate with the client's Connection class?  Also in
src/tests I've noticed that "rdma" is an option for the ConnectionOptions
class.  How will this class fit into the scheme of things?

Thanks in advance for any info,

Greg Marsh
Network-Based Computing Lab
Ohio State University
marshgr@cse.ohio-state.edu
http://nowlab.cse.ohio-state.edu/

Re: What are Qpid's plans for RDMA?

[Andrew Stitcher <as...@redhat.com>]

On Thu, 2008-09-25 at 20:20 -0400, gregory james marsh wrote:
> 
> Hello,
> 
> I'm with the Network-Based Computing Lab at Ohio State University.  Our
> lab primarily does research with the Message Passing Interface (MPI) over
> interconnects like InfiniBand and RDMA.  Recently we have started
> evaluating/testing Qpid with the TCP adaptations to InfiniBand (Sockets
> Direct Protocol [SDP] and IP over InfiniBand [IPoIB]).
> 
> We would like to know more about where the Qpid project plans to go with
> RDMA.  Could anyone give me a brief overview and anticipated timeline of
> the planned functionality?

As of svn revision 697101 The RDMA support in trunk qpid is pretty
stable so if you are using a trunk version more recent than that you
have RDMA support.

The more recent changes have been performance/scalability related.

RDMA is implemented as 2 protocol plugins one for the broker side
(rdma.so) and one for the client side (rdmaconnector.so). In an
installed broker/client library the modules will be found/loaded
automatically. If compiling from scratch the GNU defaults are used and
the modules end up in /usr/local/lib/qpid/daemon
& /usr/local/lib/qpid/client.

However if you are running from within the development tree you'll need
--load-module <path to rdma.so> added to your broker command line. And
you'll need something like:

export QPID_LOAD_MODULE=<path to rdmaconnector.so>

before you run your client code to enable the client library to find the
rdma module.

If you are using the test client in the qpid source tree you can add -P
rdma (or --protocol rdma) to tell them to use rdma instead of tcp. As we
use rdmacm for connection management Infiniband connections need to be
running IPoIB with an assigned IP address as this is used by rdmacm.

>   I've noticed some RDMA-related code submissions to the C++ trunk
> this month by Andrew Stitcher.  I've been able to run the C++ broker
> listening for RDMA via the RdmaIOPlugin class (at least according to the
> broker log).  However, how does the client side work?  Does the
> RdmaConnector class integrate with the client's Connection class?  Also in
> src/tests I've noticed that "rdma" is an option for the ConnectionOptions
> class.  How will this class fit into the scheme of things?

In your own client code you can set ConnectionOptions.protocol to "rdma"
and provide it to Connection.open to get an rdma connection.

I hope this gives you the info you need.

Andrew

Re: What are Qpid's plans for RDMA?

[Carl Trieloff <cc...@redhat.com>]

The client loads the RDMA plug in via a module loader. To try it out use 
perftest or latencytest

use the option:  -P [ --protocol ] tcp|rdma

on the broker, use --load-module to load the RMDA module.

these are both single file clients, so you can look what is needed to 
update any qpid C++ client for
RDMA by looking at these clients.

The RDMA support is native OFED, broker can listen on more than one 
transport at the same time.

RDMA will be included in the M4 release, code freeze Oct 22 I believe is 
the date M4 close down. release
then after that once committers are happy with it.

Carl.
 
> Hello,
>
> I'm with the Network-Based Computing Lab at Ohio State University.  Our
> lab primarily does research with the Message Passing Interface (MPI) over
> interconnects like InfiniBand and RDMA.  Recently we have started
> evaluating/testing Qpid with the TCP adaptations to InfiniBand (Sockets
> Direct Protocol [SDP] and IP over InfiniBand [IPoIB]).
>
> We would like to know more about where the Qpid project plans to go with
> RDMA.  Could anyone give me a brief overview and anticipated timeline of
> the planned functionality?  I've noticed some RDMA-related code submissions to the C++ trunk
> this month by Andrew Stitcher.  I've been able to run the C++ broker
> listening for RDMA via the RdmaIOPlugin class (at least according to the
> broker log).  However, how does the client side work?  Does the
> RdmaConnector class integrate with the client's Connection class?  Also in
> src/tests I've noticed that "rdma" is an option for the ConnectionOptions
> class.  How will this class fit into the scheme of things?
>
> Thanks in advance for any info,
>
> Greg Marsh
> Network-Based Computing Lab
> Ohio State University
> marshgr@cse.ohio-state.edu
> http://nowlab.cse.ohio-state.edu/
>
>
>
>
>