RDMA buffers and maxFrameSize

[gregory james marsh <ma...@cse.ohio-state.edu>]

Hi,

I'm looking into the use of Infiniband send/recv Buffers (ibv_reg_mr) in 
the C++ RDMA connector code (M4 RC3).  As far as I can tell, the buffer 
size is based on the ConnectionSettings.maxFrameSize specified by the 
client via the following object chain:

ConnectionSettings.maxFrameSize
|
RdmaConnector
|
ConnectionParams.maxRecvBufferSize
|
AsynchIO.bufferSize
|
createBuffer(bufferSize)

I've noticed that default maxFrameSize is 65535 and clients can set it 
LOWER without any problem.  I tried setting it to 200,000 just to see what 
would happen.  It appears the code iteratively tries to reduce large 
maxFrameSizes to something below 65535:

$ grep createBuffer broker_trace.txt  // My own tracing
rdma_wrap.h:  QueuePair:  createBuffer(3392)
rdma_wrap.h:  QueuePair:  createBuffer(3392)
.
.
$ grep "createBuffer(3392)" broker_trace.txt | wc -l
128  // 64 each for send & recv


Here is the calculation that appears to happen:

3392 = 200,000 - (65536 * 3)

This appears to be over compensation for too large of maxFrameSize.  Could 
someone point me to the file in the code base where this adjustment is 
happening, I'd like to experiment a bit.

Thanks,

Gregory Marsh

Re: RDMA buffers and maxFrameSize

[Gordon Sim <gs...@redhat.com>]

gregory james marsh wrote:
> 
> 
> Hi,
> 
> I'm looking into the use of Infiniband send/recv Buffers (ibv_reg_mr) in 
> the C++ RDMA connector code (M4 RC3).  As far as I can tell, the buffer 
> size is based on the ConnectionSettings.maxFrameSize specified by the 
> client via the following object chain:
> 
> ConnectionSettings.maxFrameSize
> |
> RdmaConnector
> |
> ConnectionParams.maxRecvBufferSize
> |
> AsynchIO.bufferSize
> |
> createBuffer(bufferSize)
> 
> I've noticed that default maxFrameSize is 65535 and clients can set it 
> LOWER without any problem.  I tried setting it to 200,000 just to see 
> what would happen.  It appears the code iteratively tries to reduce 
> large maxFrameSizes to something below 65535:
> 
> $ grep createBuffer broker_trace.txt  // My own tracing
> rdma_wrap.h:  QueuePair:  createBuffer(3392)
> rdma_wrap.h:  QueuePair:  createBuffer(3392)

The maximum frame size is a 16 bit value so 65535 bytes is the maximum 
value it can take. The 3392 value is the result of truncating 200000 to 
16 bits.