You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@crail.apache.org by David Crespi <da...@storedgesystems.com> on 2019/06/25 18:09:56 UTC

Question on crail type-1 with terasort shuffle data

I’m starting to run terasort now with the shuffle data going to crail type-1.
I’ve got the type-2 set to nqn: nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
which appears to be working okay for the result data.  When I introduce type-1 into my config,
it looks like the namenode gets confused and picks the nqn of the type-2 crailstore, instead of
the one it is assigned (bolded in red below).

In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and then 5 type-1 added to the
spark worker nodes (192.168.3.101-105).  All are running the same image as the namenode,
which is the spark master.

Is there some additional setting I need to include for this?
These are the variables that get set for each container (added via environment)
# NVMf storage
crail.storage.nvmf.ip                     $NVMF_IP
crail.storage.nvmf.port                 $NVMF_PORT
crail.storage.nvmf.nqn                  $NVMF_NQN
crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
crail.storage.nvmf.namespace      $NVMF_NAMESPACE

19/06/25 10:46:43 INFO crail: creating singleton crail file system
19/06/25 10:46:43 INFO crail: crail.version 3101
19/06/25 10:46:43 INFO crail: crail.directorydepth 16
19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
19/06/25 10:46:43 INFO crail: crail.cachelimit 0
19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
19/06/25 10:46:43 INFO crail: crail.user crail
19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
19/06/25 10:46:43 INFO crail: crail.debug true
19/06/25 10:46:43 INFO crail: crail.statistics true
19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
19/06/25 10:46:43 INFO crail: crail.slicesize 65536
19/06/25 10:46:43 INFO crail: crail.singleton true
19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
19/06/25 10:46:43 INFO crail: crail.cacheimpl org.apache.crail.memory.MappedBufferCache
19/06/25 10:46:43 INFO crail: crail.locationmap
19/06/25 10:46:43 INFO crail: crail.namenode.address crail://192.168.1.164:9060
19/06/25 10:46:43 INFO crail: crail.namenode.blockselection roundrobin
19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
19/06/25 10:46:43 INFO crail: crail.namenode.rpctype org.apache.crail.namenode.rpc.tcp.TcpNameNode
19/06/25 10:46:43 INFO crail: crail.namenode.log
19/06/25 10:46:43 INFO crail: crail.storage.types org.apache.crail.storage.nvmf.NvmfStorageTier
19/06/25 10:46:43 INFO crail: crail.storage.classes 2
19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0, bufferCount 1024
19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn nqn.2018-12.com.StorEdgeSystems:cntlr13
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize 1073741824
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0, queueDepth 32, messageSize 512, nodealy true
19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
19/06/25 10:46:43 INFO crail: connected to namenode(s) /192.168.1.164:9060
19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, streamId 1, isDir true, writeHint 0
19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport address = /192.168.3.100:4420, subsystem NQN = nqn.2018-12.com.StorEdgeSystems:cntlr13
19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
19/06/25 10:46:43 INFO disni: jverbs jni version 32
19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs size 28, native size 16
19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32, native size 32
19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size 72, native size 128
19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48, native size 48
19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16, native size 16
19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size 40, native size 40
19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48, native size 48
19/06/25 10:46:43 INFO disni: createEventChannel, objId 139964194703408
19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
19/06/25 10:46:43 INFO disni: createId, id 139964194793600
19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
19/06/25 10:46:43 INFO disni: resolveAddr, addres /192.168.3.100:4420
19/06/25 10:46:43 INFO disni: resolveRoute, id 0
19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
19/06/25 10:46:43 INFO disni: setting up protection domain, context 461, pd 1
19/06/25 10:46:43 INFO disni: new endpoint CQ processor
19/06/25 10:46:43 INFO disni: createCompChannel, context 139962246807056
19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe 64
19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176, send_wr size 32, recv_wr_size 32
19/06/25 10:46:43 INFO disni: connect, id 0
19/06/25 10:46:43 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress /192.168.3.100:4420
19/06/25 10:46:43 INFO disni: createId, id 139964195036000
19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
19/06/25 10:46:43 INFO disni: resolveAddr, addres /192.168.3.100:4420
19/06/25 10:46:43 INFO disni: resolveRoute, id 0
19/06/25 10:46:43 INFO disni: setting up protection domain, context 461, pd 1
19/06/25 10:46:43 INFO disni: new endpoint CQ processor
19/06/25 10:46:43 INFO disni: createCompChannel, context 139962246807056
19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe 128
19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304, send_wr size 64, recv_wr_size 64
19/06/25 10:46:43 INFO disni: connect, id 0
19/06/25 10:46:43 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress /192.168.3.100:4420
19/06/25 10:46:43 INFO crail: EndpointCache miss /192.168.3.100:4420, fsId 0, cache size 1
19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true, success
19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, streamId 1, capacity 262656
19/06/25 10:46:43 INFO crail: createNode: name /spark, type DIRECTORY, storageAffinity 0, locationAffinity 0
19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, streamId 2, isDir true, writeHint 0
19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420, fsId 0
19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd 4, token 0
19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, streamId 2, capacity 524800
19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast, type DIRECTORY, storageAffinity 0, locationAffinity 0
19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark, fd 4, streamId 3, isDir true, writeHint 0
19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport address = /192.168.3.104:4420, subsystem NQN = nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be nqn.2018-12.com.StorEdgeSystems:worker-4)
19/06/25 10:46:43 INFO disni: createEventChannel, objId 139964195131168
19/06/25 10:46:43 INFO disni: createId, id 139964195141520
19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
19/06/25 10:46:43 INFO disni: resolveAddr, addres /192.168.3.104:4420
19/06/25 10:46:43 INFO disni: resolveRoute, id 0
19/06/25 10:46:43 INFO disni: setting up protection domain, context 461, pd 1
19/06/25 10:46:43 INFO disni: new endpoint CQ processor
19/06/25 10:46:43 INFO disni: createCompChannel, context 139962246807056
19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe 64
19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896, send_wr size 32, recv_wr_size 32
19/06/25 10:46:43 INFO disni: connect, id 0
19/06/25 10:46:43 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress /192.168.3.104:4420
19/06/25 10:46:43 INFO crail: ERROR: failed data operation
com.ibm.jnvmf.UnsuccessfulComandException: Command was not successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 - The host is not allowed to establish an association to any controller in the NVM subsystem or the host is not allowed to establish an association to the specified controller., CID: 0, Do_not_retry: false, More: false, SQHD: 0}
        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
        at com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)


Regards,

           David

Re: Question on crail type-1 with terasort shuffle data

Posted by Jonas Pfefferle <pe...@japf.ch>.

I'm talking subsystem NQN. Host NQN should not be relevant. Per default Host 
NQN of the clients is generated randomly (you can set it via the 
configuration file).


  On Wed, 26 Jun 2019 16:22:18 +0000
  David Crespi <da...@storedgesystems.com> wrote:
> We’re talking host nqn right, not subsystem nqn?
> 
> David
> 714-476-2692
> 
>> On Jun 26, 2019, at 09:17, Jonas Pfefferle <pe...@japf.ch> wrote:
>> 
>> There can only be 1 NQN in the whole system. For multiple storage 
>>classes just use different ip/port combinations with the same NQN. 
>>e.g. start a NVMf server on node 1 with storage class 1 with NQN X 
>>and start a server on node 2 with storage class 2 with NQN X. You can 
>>start another server on node 1 by choosing a different port but still 
>>use the same NQN X.
>> I know that this is cumbersome to use. We are working on it, we just 
>>have not found a solution that makes everyone happy :-)
>> 
>> 
>> Regards,
>> Jonas
>> 
>> On Wed, 26 Jun 2019 16:09:06 +0000
>> David Crespi <da...@storedgesystems.com> wrote:
>>> Sorry Jonas but I’m still confused.
>>> I started off doing this by having each storage class 1 connected to 
>>>it’s own subsystem,
>>> with a single namespace.  This is where I was seeing the correct IP 
>>>address of that subsystem,
>>> but the wrong controller name.  It was the controller name of the 
>>>storage class 2, that was also
>>> registered with the namenode.  So this is the correct way of using 
>>>it, but it’s having a problem
>>> determining which controller to identify?
>>> Regards,
>>>          David
>>> From: Jonas Pfefferle<ma...@japf.ch>
>>> Sent: Wednesday, June 26, 2019 9:01 AM
>>> To: dev@crail.apache.org<ma...@crail.apache.org>; David 
>>>Crespi<ma...@storedgesystems.com>
>>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>> I'm sure you already figured this out but just to make sure we are 
>>>on the
>>> same page. The NVMf storage tier (or datanode, we use them 
>>>interchangeably)
>>> is just another NVMf client that connects to the NVMf server and 
>>>retrieves
>>> all necessary information like capacity etc. to register the 
>>>datanode to the
>>> namenode. When starting a NVMf datanode you can specify via the 
>>>commandline
>>> what storage class the datanode should be in. It then registers the 
>>>ip/port
>>> combination with the storage class information at the namenode. Now 
>>>whenever
>>> a block is requested from the namenode by a client, the namenode 
>>>sends the
>>> ip/port information and the storage class to the client. With this
>>> information the client connects to the NVMf server (not the 
>>>datanode)
>>> directly with the NQN provided in the config file, namespace 1 and 
>>>the
>>> ip/port information it got from the namenode.
>>> I hope this makes it clearer.
>>> Regards,
>>> Jonas
>>> On Wed, 26 Jun 2019 14:55:05 +0000
>>> David Crespi <da...@storedgesystems.com> wrote:
>>>> Yep.  I wanted to use crail-store for temp results storage, storage
>>>> class 2, AND the shuffle data, with
>>>> 
>>>> storage class 1.
>>>> 
>>>> 
>>>> Regards,
>>>> 
>>>> 
>>>>          David
>>>> 
>>>> 
>>>> ________________________________
>>>> From: Jonas Pfefferle <pe...@japf.ch>
>>>> Sent: Wednesday, June 26, 2019 7:51:43 AM
>>>> To: dev@crail.apache.org; David Crespi
>>>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>>> 
>>>> David,
>>>> 
>>>> 
>>>> Let's make sure we talk about the same thing, I assume with type 1
>>>> and 2 you
>>>> mean storage classes?
>>>> 
>>>> Regards,
>>>> Jonas
>>>> 
>>>> On Wed, 26 Jun 2019 14:25:52 +0000
>>>> David Crespi <da...@storedgesystems.com> wrote:
>>>>> I’m a bit confused now. If it only ever handles 1 namespace, then
>>>>> multiple datanodes are useless.
>>>>> 
>>>>> Or are you saying only 1 namespace for type-1?  If you’re not saying
>>>>> only one for type-1, how do you
>>>>> 
>>>>> mix types if limited to 1 namespace?
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> 
>>>>>          David
>>>>> 
>>>>> 
>>>>> ________________________________
>>>>> From: Jonas Pfefferle <pe...@japf.ch>
>>>>> Sent: Wednesday, June 26, 2019 12:25:38 AM
>>>>> To: dev@crail.apache.org; David Crespi
>>>>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>>>> 
>>>>> I forgot to add: Namespace has to be 1 for the moment. This is a
>>>>> limitation
>>>>> in how the namenode handles storage tier metadata (essentially it
>>>>> only
>>>>> stores ip/port). You can set it on the server side however it will
>>>>> not be
>>>>> respected by the client, i.e. the client will always use namespace
>>>>> id 1.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> Jonas
>>>>> 
>>>>> On Wed, 26 Jun 2019 09:17:40 +0200
>>>>> "Jonas Pfefferle" <pe...@japf.ch> wrote:
>>>>>> Hi David,
>>>>>> 
>>>>>> 
>>>>>> This is currently a limitation of the NVMf storage tier. You need to
>>>>>> use the same NQN on all datanodes. The port can be different 
>>>>>>however.
>>>>>> I will clarify this in the documentation.
>>>>>> 
>>>>>> Regards,
>>>>>> Jonas
>>>>>> 
>>>>>> On Tue, 25 Jun 2019 19:43:04 +0000
>>>>>> David Crespi <da...@storedgesystems.com> wrote:
>>>>>>> It occurred to me that perhaps since I’m using the NVMf, that the
>>>>>>> namenode can only handle one
>>>>>>> 
>>>>>>> Subsystem.  And for the type-1, these resources would need to be
>>>>>>> added under the same subsystem,
>>>>>>> 
>>>>>>> but as different namespace’s.  Is this correct?  I did try this and
>>>>>>> it appears to be working. If this IS the
>>>>>>> 
>>>>>>> correct way of utilizing these, how do I know which namespace(s) are
>>>>>>> being accessed?  I don’t see
>>>>>>> 
>>>>>>> anything in the logs that identify which is being addressed.
>>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> 
>>>>>>>          David
>>>>>>> 
>>>>>>> 
>>>>>>> ________________________________
>>>>>>> From: David Crespi <da...@storedgesystems.com>
>>>>>>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>>>>>>> To: dev@crail.apache.org
>>>>>>> Subject: Question on crail type-1 with terasort shuffle data
>>>>>>> 
>>>>>>> I’m starting to run terasort now with the shuffle data going to
>>>>>>> crail type-1.
>>>>>>> I’ve got the type-2 set to nqn:
>>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>>>>>>> which appears to be working okay for the result data.  When I
>>>>>>> introduce type-1 into my config,
>>>>>>> it looks like the namenode gets confused and picks the nqn of the
>>>>>>> type-2 crailstore, instead of
>>>>>>> the one it is assigned (bolded in red below).
>>>>>>> 
>>>>>>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
>>>>>>> then 5 type-1 added to the
>>>>>>> spark worker nodes (192.168.3.101-105).  All are running the same
>>>>>>> image as the namenode,
>>>>>>> which is the spark master.
>>>>>>> 
>>>>>>> Is there some additional setting I need to include for this?
>>>>>>> These are the variables that get set for each container (added via
>>>>>>> environment)
>>>>>>> # NVMf storage
>>>>>>> crail.storage.nvmf.ip                     $NVMF_IP
>>>>>>> crail.storage.nvmf.port                 $NVMF_PORT
>>>>>>> crail.storage.nvmf.nqn                  $NVMF_NQN
>>>>>>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>>>>>>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>>>>>>> 
>>>>>>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.user crail
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.debug true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>>>>>> org.apache.crail.memory.MappedBufferCache
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>>>>>> crail://192.168.1.164:9060
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>>>>>> roundrobin
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>>>>>> org.apache.crail.namenode.rpc.tcp.TcpNameNode
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>>>>>> org.apache.crail.storage.nvmf.NvmfStorageTier
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>>>>>>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>>>>>> bufferCount 1024
>>>>>>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>>>>>> nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>>>>>> 1073741824
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>>>>>>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>>>>>> queueDepth 32, messageSize 512, nodealy true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>>>>>>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>>>>>> /192.168.1.164:9060
>>>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>>>>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>>>>>>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>>>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>>>> streamId 1, isDir true, writeHint 0
>>>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>>>> address = /192.168.3.100:4420, subsystem NQN =
>>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>>>>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
>>>>>>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>>>>>>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>>>>>> size 28, native size 16
>>>>>>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
>>>>>>> native size 32
>>>>>>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
>>>>>>> 72, native size 128
>>>>>>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>>>>>> native size 48
>>>>>>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
>>>>>>> native size 16
>>>>>>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
>>>>>>> 40, native size 40
>>>>>>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
>>>>>>> native size 48
>>>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>>>> 139964194703408
>>>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>>>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>>> /192.168.3.100:4420
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>>>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>>> 461, pd 1
>>>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>>> 139962246807056
>>>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
>>>>>>> 64
>>>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>>>>>> send_wr size 32, recv_wr_size 32
>>>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, 
>>>>>>>dstAddress
>>>>>>> /192.168.3.100:4420
>>>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>>>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>>> /192.168.3.100:4420
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>>> 461, pd 1
>>>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>>> 139962246807056
>>>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
>>>>>>> 128
>>>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>>>>>> send_wr size 64, recv_wr_size 64
>>>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, 
>>>>>>>dstAddress
>>>>>>> /192.168.3.100:4420
>>>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>>>>>> /192.168.3.100:4420, fsId 0, cache size 1
>>>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
>>>>>>> success
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>>>> streamId 1, capacity 262656
>>>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>>>>>> DIRECTORY, storageAffinity 0, locationAffinity 0
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>>>> streamId 2, isDir true, writeHint 0
>>>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>>>>>> fsId 0
>>>>>>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
>>>>>>> 4, token 0
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>>>> streamId 2, capacity 524800
>>>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>>>>>> type DIRECTORY, storageAffinity 0, locationAffinity 0
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>>>>>> fd 4, streamId 3, isDir true, writeHint 0
>>>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>>>> address = /192.168.3.104:4420, subsystem NQN =
>>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
>>>>>>> nqn.2018-12.com.StorEdgeSystems:worker-4)
>>>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>>>> 139964195131168
>>>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>>>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>>>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>>> /192.168.3.104:4420
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>>> 461, pd 1
>>>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>>> 139962246807056
>>>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
>>>>>>> 64
>>>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>>>>>> send_wr size 32, recv_wr_size 32
>>>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, 
>>>>>>>dstAddress
>>>>>>> /192.168.3.104:4420
>>>>>>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>>>>>>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>>>>>> successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
>>>>>>> The host is not allowed to establish an association to any 
>>>>>>>controller
>>>>>>> in the NVM subsystem or the host is not allowed to establish an
>>>>>>> association to the specified controller., CID: 0, Do_not_retry:
>>>>>>> false, More: false, SQHD: 0}
>>>>>>>       at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>>>>>>       at
>>>>>>> com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>>          David
>>>>>>> 
>>>>>>

Re: Question on crail type-1 with terasort shuffle data

Posted by David Crespi <da...@storedgesystems.com>.

We’re talking host nqn right, not subsystem nqn?

David
714-476-2692

> On Jun 26, 2019, at 09:17, Jonas Pfefferle <pe...@japf.ch> wrote:
> 
> There can only be 1 NQN in the whole system. For multiple storage classes just use different ip/port combinations with the same NQN. e.g. start a NVMf server on node 1 with storage class 1 with NQN X and start a server on node 2 with storage class 2 with NQN X. You can start another server on node 1 by choosing a different port but still use the same NQN X.
> I know that this is cumbersome to use. We are working on it, we just have not found a solution that makes everyone happy :-)
> 
> 
> Regards,
> Jonas
> 
> On Wed, 26 Jun 2019 16:09:06 +0000
> David Crespi <da...@storedgesystems.com> wrote:
>> Sorry Jonas but I’m still confused.
>> I started off doing this by having each storage class 1 connected to it’s own subsystem,
>> with a single namespace.  This is where I was seeing the correct IP address of that subsystem,
>> but the wrong controller name.  It was the controller name of the storage class 2, that was also
>> registered with the namenode.  So this is the correct way of using it, but it’s having a problem
>> determining which controller to identify?
>> Regards,
>>          David
>> From: Jonas Pfefferle<ma...@japf.ch>
>> Sent: Wednesday, June 26, 2019 9:01 AM
>> To: dev@crail.apache.org<ma...@crail.apache.org>; David Crespi<ma...@storedgesystems.com>
>> Subject: Re: Question on crail type-1 with terasort shuffle data
>> I'm sure you already figured this out but just to make sure we are on the
>> same page. The NVMf storage tier (or datanode, we use them interchangeably)
>> is just another NVMf client that connects to the NVMf server and retrieves
>> all necessary information like capacity etc. to register the datanode to the
>> namenode. When starting a NVMf datanode you can specify via the commandline
>> what storage class the datanode should be in. It then registers the ip/port
>> combination with the storage class information at the namenode. Now whenever
>> a block is requested from the namenode by a client, the namenode sends the
>> ip/port information and the storage class to the client. With this
>> information the client connects to the NVMf server (not the datanode)
>> directly with the NQN provided in the config file, namespace 1 and the
>> ip/port information it got from the namenode.
>> I hope this makes it clearer.
>> Regards,
>> Jonas
>> On Wed, 26 Jun 2019 14:55:05 +0000
>> David Crespi <da...@storedgesystems.com> wrote:
>>> Yep.  I wanted to use crail-store for temp results storage, storage
>>> class 2, AND the shuffle data, with
>>> 
>>> storage class 1.
>>> 
>>> 
>>> Regards,
>>> 
>>> 
>>>          David
>>> 
>>> 
>>> ________________________________
>>> From: Jonas Pfefferle <pe...@japf.ch>
>>> Sent: Wednesday, June 26, 2019 7:51:43 AM
>>> To: dev@crail.apache.org; David Crespi
>>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>> 
>>> David,
>>> 
>>> 
>>> Let's make sure we talk about the same thing, I assume with type 1
>>> and 2 you
>>> mean storage classes?
>>> 
>>> Regards,
>>> Jonas
>>> 
>>> On Wed, 26 Jun 2019 14:25:52 +0000
>>> David Crespi <da...@storedgesystems.com> wrote:
>>>> I’m a bit confused now. If it only ever handles 1 namespace, then
>>>> multiple datanodes are useless.
>>>> 
>>>> Or are you saying only 1 namespace for type-1?  If you’re not saying
>>>> only one for type-1, how do you
>>>> 
>>>> mix types if limited to 1 namespace?
>>>> 
>>>> 
>>>> Regards,
>>>> 
>>>> 
>>>>          David
>>>> 
>>>> 
>>>> ________________________________
>>>> From: Jonas Pfefferle <pe...@japf.ch>
>>>> Sent: Wednesday, June 26, 2019 12:25:38 AM
>>>> To: dev@crail.apache.org; David Crespi
>>>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>>> 
>>>> I forgot to add: Namespace has to be 1 for the moment. This is a
>>>> limitation
>>>> in how the namenode handles storage tier metadata (essentially it
>>>> only
>>>> stores ip/port). You can set it on the server side however it will
>>>> not be
>>>> respected by the client, i.e. the client will always use namespace
>>>> id 1.
>>>> 
>>>> 
>>>> Regards,
>>>> Jonas
>>>> 
>>>> On Wed, 26 Jun 2019 09:17:40 +0200
>>>> "Jonas Pfefferle" <pe...@japf.ch> wrote:
>>>>> Hi David,
>>>>> 
>>>>> 
>>>>> This is currently a limitation of the NVMf storage tier. You need to
>>>>> use the same NQN on all datanodes. The port can be different however.
>>>>> I will clarify this in the documentation.
>>>>> 
>>>>> Regards,
>>>>> Jonas
>>>>> 
>>>>> On Tue, 25 Jun 2019 19:43:04 +0000
>>>>> David Crespi <da...@storedgesystems.com> wrote:
>>>>>> It occurred to me that perhaps since I’m using the NVMf, that the
>>>>>> namenode can only handle one
>>>>>> 
>>>>>> Subsystem.  And for the type-1, these resources would need to be
>>>>>> added under the same subsystem,
>>>>>> 
>>>>>> but as different namespace’s.  Is this correct?  I did try this and
>>>>>> it appears to be working. If this IS the
>>>>>> 
>>>>>> correct way of utilizing these, how do I know which namespace(s) are
>>>>>> being accessed?  I don’t see
>>>>>> 
>>>>>> anything in the logs that identify which is being addressed.
>>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> 
>>>>>>          David
>>>>>> 
>>>>>> 
>>>>>> ________________________________
>>>>>> From: David Crespi <da...@storedgesystems.com>
>>>>>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>>>>>> To: dev@crail.apache.org
>>>>>> Subject: Question on crail type-1 with terasort shuffle data
>>>>>> 
>>>>>> I’m starting to run terasort now with the shuffle data going to
>>>>>> crail type-1.
>>>>>> I’ve got the type-2 set to nqn:
>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>>>>>> which appears to be working okay for the result data.  When I
>>>>>> introduce type-1 into my config,
>>>>>> it looks like the namenode gets confused and picks the nqn of the
>>>>>> type-2 crailstore, instead of
>>>>>> the one it is assigned (bolded in red below).
>>>>>> 
>>>>>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
>>>>>> then 5 type-1 added to the
>>>>>> spark worker nodes (192.168.3.101-105).  All are running the same
>>>>>> image as the namenode,
>>>>>> which is the spark master.
>>>>>> 
>>>>>> Is there some additional setting I need to include for this?
>>>>>> These are the variables that get set for each container (added via
>>>>>> environment)
>>>>>> # NVMf storage
>>>>>> crail.storage.nvmf.ip                     $NVMF_IP
>>>>>> crail.storage.nvmf.port                 $NVMF_PORT
>>>>>> crail.storage.nvmf.nqn                  $NVMF_NQN
>>>>>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>>>>>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>>>>>> 
>>>>>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>>>>>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>>>>>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>>>>>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>>>>>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>>>>>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>>>>>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>>>>>> 19/06/25 10:46:43 INFO crail: crail.user crail
>>>>>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>>>>>> 19/06/25 10:46:43 INFO crail: crail.debug true
>>>>>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>>>>>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>>>>>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>>>>>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>>>>>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>>>>>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>>>>>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>>>>>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>>>>> org.apache.crail.memory.MappedBufferCache
>>>>>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>>>>> crail://192.168.1.164:9060
>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>>>>> roundrobin
>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>>>>> org.apache.crail.namenode.rpc.tcp.TcpNameNode
>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>>>>> org.apache.crail.storage.nvmf.NvmfStorageTier
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>>>>>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>>>>> bufferCount 1024
>>>>>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>>>>> nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>>>>> 1073741824
>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>>>>>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>>>>> queueDepth 32, messageSize 512, nodealy true
>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>>>>>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>>>>> /192.168.1.164:9060
>>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>>>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>>>>>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>>> streamId 1, isDir true, writeHint 0
>>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>>> address = /192.168.3.100:4420, subsystem NQN =
>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>>>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
>>>>>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>>>>>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>>>>> size 28, native size 16
>>>>>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
>>>>>> native size 32
>>>>>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
>>>>>> 72, native size 128
>>>>>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>>>>> native size 48
>>>>>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
>>>>>> native size 16
>>>>>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
>>>>>> 40, native size 40
>>>>>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
>>>>>> native size 48
>>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>>> 139964194703408
>>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>> /192.168.3.100:4420
>>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>> 461, pd 1
>>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>> 139962246807056
>>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
>>>>>> 64
>>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>>>>> send_wr size 32, recv_wr_size 32
>>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress
>>>>>> /192.168.3.100:4420
>>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>> /192.168.3.100:4420
>>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>> 461, pd 1
>>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>> 139962246807056
>>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
>>>>>> 128
>>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>>>>> send_wr size 64, recv_wr_size 64
>>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress
>>>>>> /192.168.3.100:4420
>>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>>>>> /192.168.3.100:4420, fsId 0, cache size 1
>>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
>>>>>> success
>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>>> streamId 1, capacity 262656
>>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>>>>> DIRECTORY, storageAffinity 0, locationAffinity 0
>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>>> streamId 2, isDir true, writeHint 0
>>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>>>>> fsId 0
>>>>>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
>>>>>> 4, token 0
>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>>> streamId 2, capacity 524800
>>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>>>>> type DIRECTORY, storageAffinity 0, locationAffinity 0
>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>>>>> fd 4, streamId 3, isDir true, writeHint 0
>>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>>> address = /192.168.3.104:4420, subsystem NQN =
>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
>>>>>> nqn.2018-12.com.StorEdgeSystems:worker-4)
>>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>>> 139964195131168
>>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>> /192.168.3.104:4420
>>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>> 461, pd 1
>>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>> 139962246807056
>>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
>>>>>> 64
>>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>>>>> send_wr size 32, recv_wr_size 32
>>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress
>>>>>> /192.168.3.104:4420
>>>>>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>>>>>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>>>>> successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
>>>>>> The host is not allowed to establish an association to any controller
>>>>>> in the NVM subsystem or the host is not allowed to establish an
>>>>>> association to the specified controller., CID: 0, Do_not_retry:
>>>>>> false, More: false, SQHD: 0}
>>>>>>       at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>>>>>       at
>>>>>> com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>>>>> 
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>>          David
>>>>>> 
>>>>> 
>

Re: Question on crail type-1 with terasort shuffle data

Posted by Jonas Pfefferle <pe...@japf.ch>.

There can only be 1 NQN in the whole system. For multiple storage classes 
just use different ip/port combinations with the same NQN. e.g. start a NVMf 
server on node 1 with storage class 1 with NQN X and start a server on node 
2 with storage class 2 with NQN X. You can start another server on node 1 by 
choosing a different port but still use the same NQN X.
I know that this is cumbersome to use. We are working on it, we just have 
not found a solution that makes everyone happy :-)


Regards,
Jonas

  On Wed, 26 Jun 2019 16:09:06 +0000
  David Crespi <da...@storedgesystems.com> wrote:
> Sorry Jonas but I’m still confused.
> I started off doing this by having each storage class 1 connected to 
>it’s own subsystem,
> with a single namespace.  This is where I was seeing the correct IP 
>address of that subsystem,
> but the wrong controller name.  It was the controller name of the 
>storage class 2, that was also
> registered with the namenode.  So this is the correct way of using 
>it, but it’s having a problem
> determining which controller to identify?
> 
> Regards,
> 
>           David
> 
>From: Jonas Pfefferle<ma...@japf.ch>
> Sent: Wednesday, June 26, 2019 9:01 AM
> To: dev@crail.apache.org<ma...@crail.apache.org>; David 
>Crespi<ma...@storedgesystems.com>
> Subject: Re: Question on crail type-1 with terasort shuffle data
> 
> I'm sure you already figured this out but just to make sure we are 
>on the
> same page. The NVMf storage tier (or datanode, we use them 
>interchangeably)
> is just another NVMf client that connects to the NVMf server and 
>retrieves
> all necessary information like capacity etc. to register the 
>datanode to the
> namenode. When starting a NVMf datanode you can specify via the 
>commandline
> what storage class the datanode should be in. It then registers the 
>ip/port
> combination with the storage class information at the namenode. Now 
>whenever
> a block is requested from the namenode by a client, the namenode 
>sends the
> ip/port information and the storage class to the client. With this
> information the client connects to the NVMf server (not the 
>datanode)
> directly with the NQN provided in the config file, namespace 1 and 
>the
> ip/port information it got from the namenode.
> 
> 
> I hope this makes it clearer.
> 
> Regards,
> Jonas
> 
>  On Wed, 26 Jun 2019 14:55:05 +0000
>  David Crespi <da...@storedgesystems.com> wrote:
>> Yep.  I wanted to use crail-store for temp results storage, storage
>>class 2, AND the shuffle data, with
>>
>> storage class 1.
>>
>>
>> Regards,
>>
>>
>>           David
>>
>>
>> ________________________________
>>From: Jonas Pfefferle <pe...@japf.ch>
>> Sent: Wednesday, June 26, 2019 7:51:43 AM
>> To: dev@crail.apache.org; David Crespi
>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>
>> David,
>>
>>
>> Let's make sure we talk about the same thing, I assume with type 1
>>and 2 you
>> mean storage classes?
>>
>> Regards,
>> Jonas
>>
>>  On Wed, 26 Jun 2019 14:25:52 +0000
>>  David Crespi <da...@storedgesystems.com> wrote:
>>> I’m a bit confused now. If it only ever handles 1 namespace, then
>>>multiple datanodes are useless.
>>>
>>> Or are you saying only 1 namespace for type-1?  If you’re not saying
>>>only one for type-1, how do you
>>>
>>> mix types if limited to 1 namespace?
>>>
>>>
>>> Regards,
>>>
>>>
>>>           David
>>>
>>>
>>> ________________________________
>>>From: Jonas Pfefferle <pe...@japf.ch>
>>> Sent: Wednesday, June 26, 2019 12:25:38 AM
>>> To: dev@crail.apache.org; David Crespi
>>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>>
>>> I forgot to add: Namespace has to be 1 for the moment. This is a
>>>limitation
>>> in how the namenode handles storage tier metadata (essentially it
>>>only
>>> stores ip/port). You can set it on the server side however it will
>>>not be
>>> respected by the client, i.e. the client will always use namespace
>>>id 1.
>>>
>>>
>>> Regards,
>>> Jonas
>>>
>>>  On Wed, 26 Jun 2019 09:17:40 +0200
>>>  "Jonas Pfefferle" <pe...@japf.ch> wrote:
>>>> Hi David,
>>>>
>>>>
>>>> This is currently a limitation of the NVMf storage tier. You need to
>>>>use the same NQN on all datanodes. The port can be different however.
>>>>I will clarify this in the documentation.
>>>>
>>>> Regards,
>>>> Jonas
>>>>
>>>>  On Tue, 25 Jun 2019 19:43:04 +0000
>>>>  David Crespi <da...@storedgesystems.com> wrote:
>>>>> It occurred to me that perhaps since I’m using the NVMf, that the
>>>>>namenode can only handle one
>>>>>
>>>>> Subsystem.  And for the type-1, these resources would need to be
>>>>>added under the same subsystem,
>>>>>
>>>>> but as different namespace’s.  Is this correct?  I did try this and
>>>>>it appears to be working. If this IS the
>>>>>
>>>>> correct way of utilizing these, how do I know which namespace(s) are
>>>>>being accessed?  I don’t see
>>>>>
>>>>> anything in the logs that identify which is being addressed.
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>>           David
>>>>>
>>>>>
>>>>> ________________________________
>>>>>From: David Crespi <da...@storedgesystems.com>
>>>>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>>>>> To: dev@crail.apache.org
>>>>> Subject: Question on crail type-1 with terasort shuffle data
>>>>>
>>>>> I’m starting to run terasort now with the shuffle data going to
>>>>>crail type-1.
>>>>> I’ve got the type-2 set to nqn:
>>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>>>>> which appears to be working okay for the result data.  When I
>>>>>introduce type-1 into my config,
>>>>> it looks like the namenode gets confused and picks the nqn of the
>>>>>type-2 crailstore, instead of
>>>>> the one it is assigned (bolded in red below).
>>>>>
>>>>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
>>>>>then 5 type-1 added to the
>>>>> spark worker nodes (192.168.3.101-105).  All are running the same
>>>>>image as the namenode,
>>>>> which is the spark master.
>>>>>
>>>>> Is there some additional setting I need to include for this?
>>>>> These are the variables that get set for each container (added via
>>>>>environment)
>>>>> # NVMf storage
>>>>> crail.storage.nvmf.ip                     $NVMF_IP
>>>>> crail.storage.nvmf.port                 $NVMF_PORT
>>>>> crail.storage.nvmf.nqn                  $NVMF_NQN
>>>>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>>>>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>>>>>
>>>>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>>>>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>>>>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>>>>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>>>>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>>>>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>>>>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>>>>> 19/06/25 10:46:43 INFO crail: crail.user crail
>>>>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>>>>> 19/06/25 10:46:43 INFO crail: crail.debug true
>>>>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>>>>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>>>>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>>>>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>>>>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>>>>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>>>>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>>>>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>>>>org.apache.crail.memory.MappedBufferCache
>>>>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>>>>crail://192.168.1.164:9060
>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>>>>roundrobin
>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>>>>org.apache.crail.namenode.rpc.tcp.TcpNameNode
>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>>>>org.apache.crail.storage.nvmf.NvmfStorageTier
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>>>>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>>>>bufferCount 1024
>>>>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>>>>nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>>>>1073741824
>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>>>>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>>>>queueDepth 32, messageSize 512, nodealy true
>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>>>>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>>>>/192.168.1.164:9060
>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>>>>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>>streamId 1, isDir true, writeHint 0
>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>>address = /192.168.3.100:4420, subsystem NQN =
>>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
>>>>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>>>>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>>>>size 28, native size 16
>>>>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
>>>>>native size 32
>>>>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
>>>>>72, native size 128
>>>>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>>>>native size 48
>>>>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
>>>>>native size 16
>>>>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
>>>>>40, native size 40
>>>>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
>>>>>native size 48
>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>>139964194703408
>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>/192.168.3.100:4420
>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>461, pd 1
>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>139962246807056
>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
>>>>>64
>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>>>>send_wr size 32, recv_wr_size 32
>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress
>>>>>/192.168.3.100:4420
>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>/192.168.3.100:4420
>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>461, pd 1
>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>139962246807056
>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
>>>>>128
>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>>>>send_wr size 64, recv_wr_size 64
>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress
>>>>>/192.168.3.100:4420
>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>>>>/192.168.3.100:4420, fsId 0, cache size 1
>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
>>>>>success
>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>>streamId 1, capacity 262656
>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>>>>DIRECTORY, storageAffinity 0, locationAffinity 0
>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>>streamId 2, isDir true, writeHint 0
>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>>>>fsId 0
>>>>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
>>>>>4, token 0
>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>>streamId 2, capacity 524800
>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>>>>type DIRECTORY, storageAffinity 0, locationAffinity 0
>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>>>>fd 4, streamId 3, isDir true, writeHint 0
>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>>address = /192.168.3.104:4420, subsystem NQN =
>>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
>>>>>nqn.2018-12.com.StorEdgeSystems:worker-4)
>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>>139964195131168
>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>/192.168.3.104:4420
>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>461, pd 1
>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>139962246807056
>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
>>>>>64
>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>>>>send_wr size 32, recv_wr_size 32
>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress
>>>>>/192.168.3.104:4420
>>>>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>>>>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>>>>successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
>>>>>The host is not allowed to establish an association to any controller
>>>>>in the NVM subsystem or the host is not allowed to establish an
>>>>>association to the specified controller., CID: 0, Do_not_retry:
>>>>>false, More: false, SQHD: 0}
>>>>>        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>>>>        at
>>>>>com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>           David
>>>>>
>>>>

RE: Question on crail type-1 with terasort shuffle data

Posted by David Crespi <da...@storedgesystems.com>.

Sorry Jonas but I’m still confused.
I started off doing this by having each storage class 1 connected to it’s own subsystem,
with a single namespace.  This is where I was seeing the correct IP address of that subsystem,
but the wrong controller name.  It was the controller name of the storage class 2, that was also
registered with the namenode.  So this is the correct way of using it, but it’s having a problem
determining which controller to identify?

Regards,

           David

From: Jonas Pfefferle<ma...@japf.ch>
Sent: Wednesday, June 26, 2019 9:01 AM
To: dev@crail.apache.org<ma...@crail.apache.org>; David Crespi<ma...@storedgesystems.com>
Subject: Re: Question on crail type-1 with terasort shuffle data

I'm sure you already figured this out but just to make sure we are on the
same page. The NVMf storage tier (or datanode, we use them interchangeably)
is just another NVMf client that connects to the NVMf server and retrieves
all necessary information like capacity etc. to register the datanode to the
namenode. When starting a NVMf datanode you can specify via the commandline
what storage class the datanode should be in. It then registers the ip/port
combination with the storage class information at the namenode. Now whenever
a block is requested from the namenode by a client, the namenode sends the
ip/port information and the storage class to the client. With this
information the client connects to the NVMf server (not the datanode)
directly with the NQN provided in the config file, namespace 1 and the
ip/port information it got from the namenode.


I hope this makes it clearer.

Regards,
Jonas

  On Wed, 26 Jun 2019 14:55:05 +0000
  David Crespi <da...@storedgesystems.com> wrote:
> Yep.  I wanted to use crail-store for temp results storage, storage
>class 2, AND the shuffle data, with
>
> storage class 1.
>
>
> Regards,
>
>
>           David
>
>
> ________________________________
>From: Jonas Pfefferle <pe...@japf.ch>
> Sent: Wednesday, June 26, 2019 7:51:43 AM
> To: dev@crail.apache.org; David Crespi
> Subject: Re: Question on crail type-1 with terasort shuffle data
>
> David,
>
>
> Let's make sure we talk about the same thing, I assume with type 1
>and 2 you
> mean storage classes?
>
> Regards,
> Jonas
>
>  On Wed, 26 Jun 2019 14:25:52 +0000
>  David Crespi <da...@storedgesystems.com> wrote:
>> I’m a bit confused now. If it only ever handles 1 namespace, then
>>multiple datanodes are useless.
>>
>> Or are you saying only 1 namespace for type-1?  If you’re not saying
>>only one for type-1, how do you
>>
>> mix types if limited to 1 namespace?
>>
>>
>> Regards,
>>
>>
>>           David
>>
>>
>> ________________________________
>>From: Jonas Pfefferle <pe...@japf.ch>
>> Sent: Wednesday, June 26, 2019 12:25:38 AM
>> To: dev@crail.apache.org; David Crespi
>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>
>> I forgot to add: Namespace has to be 1 for the moment. This is a
>>limitation
>> in how the namenode handles storage tier metadata (essentially it
>>only
>> stores ip/port). You can set it on the server side however it will
>>not be
>> respected by the client, i.e. the client will always use namespace
>>id 1.
>>
>>
>> Regards,
>> Jonas
>>
>>  On Wed, 26 Jun 2019 09:17:40 +0200
>>  "Jonas Pfefferle" <pe...@japf.ch> wrote:
>>> Hi David,
>>>
>>>
>>> This is currently a limitation of the NVMf storage tier. You need to
>>>use the same NQN on all datanodes. The port can be different however.
>>>I will clarify this in the documentation.
>>>
>>> Regards,
>>> Jonas
>>>
>>>  On Tue, 25 Jun 2019 19:43:04 +0000
>>>  David Crespi <da...@storedgesystems.com> wrote:
>>>> It occurred to me that perhaps since I’m using the NVMf, that the
>>>>namenode can only handle one
>>>>
>>>> Subsystem.  And for the type-1, these resources would need to be
>>>>added under the same subsystem,
>>>>
>>>> but as different namespace’s.  Is this correct?  I did try this and
>>>>it appears to be working. If this IS the
>>>>
>>>> correct way of utilizing these, how do I know which namespace(s) are
>>>>being accessed?  I don’t see
>>>>
>>>> anything in the logs that identify which is being addressed.
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>           David
>>>>
>>>>
>>>> ________________________________
>>>>From: David Crespi <da...@storedgesystems.com>
>>>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>>>> To: dev@crail.apache.org
>>>> Subject: Question on crail type-1 with terasort shuffle data
>>>>
>>>> I’m starting to run terasort now with the shuffle data going to
>>>>crail type-1.
>>>> I’ve got the type-2 set to nqn:
>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>>>> which appears to be working okay for the result data.  When I
>>>>introduce type-1 into my config,
>>>> it looks like the namenode gets confused and picks the nqn of the
>>>>type-2 crailstore, instead of
>>>> the one it is assigned (bolded in red below).
>>>>
>>>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
>>>>then 5 type-1 added to the
>>>> spark worker nodes (192.168.3.101-105).  All are running the same
>>>>image as the namenode,
>>>> which is the spark master.
>>>>
>>>> Is there some additional setting I need to include for this?
>>>> These are the variables that get set for each container (added via
>>>>environment)
>>>> # NVMf storage
>>>> crail.storage.nvmf.ip                     $NVMF_IP
>>>> crail.storage.nvmf.port                 $NVMF_PORT
>>>> crail.storage.nvmf.nqn                  $NVMF_NQN
>>>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>>>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>>>>
>>>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>>>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>>>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>>>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>>>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>>>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>>>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>>>> 19/06/25 10:46:43 INFO crail: crail.user crail
>>>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>>>> 19/06/25 10:46:43 INFO crail: crail.debug true
>>>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>>>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>>>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>>>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>>>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>>>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>>>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>>>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>>>org.apache.crail.memory.MappedBufferCache
>>>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>>>crail://192.168.1.164:9060
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>>>roundrobin
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>>>org.apache.crail.namenode.rpc.tcp.TcpNameNode
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>>>org.apache.crail.storage.nvmf.NvmfStorageTier
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>>>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>>>bufferCount 1024
>>>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>>>nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>>>1073741824
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>>>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>>>queueDepth 32, messageSize 512, nodealy true
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>>>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>>>/192.168.1.164:9060
>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>>>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>streamId 1, isDir true, writeHint 0
>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>address = /192.168.3.100:4420, subsystem NQN =
>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
>>>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>>>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>>>size 28, native size 16
>>>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
>>>>native size 32
>>>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
>>>>72, native size 128
>>>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>>>native size 48
>>>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
>>>>native size 16
>>>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
>>>>40, native size 40
>>>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
>>>>native size 48
>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>139964194703408
>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>/192.168.3.100:4420
>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>461, pd 1
>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>139962246807056
>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
>>>>64
>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>>>send_wr size 32, recv_wr_size 32
>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress
>>>>/192.168.3.100:4420
>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>/192.168.3.100:4420
>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>461, pd 1
>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>139962246807056
>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
>>>>128
>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>>>send_wr size 64, recv_wr_size 64
>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress
>>>>/192.168.3.100:4420
>>>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>>>/192.168.3.100:4420, fsId 0, cache size 1
>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
>>>>success
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>streamId 1, capacity 262656
>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>>>DIRECTORY, storageAffinity 0, locationAffinity 0
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>streamId 2, isDir true, writeHint 0
>>>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>>>fsId 0
>>>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
>>>>4, token 0
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>streamId 2, capacity 524800
>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>>>type DIRECTORY, storageAffinity 0, locationAffinity 0
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>>>fd 4, streamId 3, isDir true, writeHint 0
>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>address = /192.168.3.104:4420, subsystem NQN =
>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
>>>>nqn.2018-12.com.StorEdgeSystems:worker-4)
>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>139964195131168
>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>/192.168.3.104:4420
>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>461, pd 1
>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>139962246807056
>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
>>>>64
>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>>>send_wr size 32, recv_wr_size 32
>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress
>>>>/192.168.3.104:4420
>>>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>>>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>>>successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
>>>>The host is not allowed to establish an association to any controller
>>>>in the NVM subsystem or the host is not allowed to establish an
>>>>association to the specified controller., CID: 0, Do_not_retry:
>>>>false, More: false, SQHD: 0}
>>>>        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>>>        at
>>>>com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>>>
>>>>
>>>> Regards,
>>>>
>>>>           David
>>>>
>>>

Re: Question on crail type-1 with terasort shuffle data

Posted by Jonas Pfefferle <pe...@japf.ch>.

I'm sure you already figured this out but just to make sure we are on the 
same page. The NVMf storage tier (or datanode, we use them interchangeably) 
is just another NVMf client that connects to the NVMf server and retrieves 
all necessary information like capacity etc. to register the datanode to the 
namenode. When starting a NVMf datanode you can specify via the commandline 
what storage class the datanode should be in. It then registers the ip/port 
combination with the storage class information at the namenode. Now whenever 
a block is requested from the namenode by a client, the namenode sends the 
ip/port information and the storage class to the client. With this 
information the client connects to the NVMf server (not the datanode) 
directly with the NQN provided in the config file, namespace 1 and the 
ip/port information it got from the namenode.


I hope this makes it clearer.

Regards,
Jonas

  On Wed, 26 Jun 2019 14:55:05 +0000
  David Crespi <da...@storedgesystems.com> wrote:
> Yep.  I wanted to use crail-store for temp results storage, storage 
>class 2, AND the shuffle data, with
> 
> storage class 1.
> 
> 
> Regards,
> 
> 
>           David
> 
> 
> ________________________________
>From: Jonas Pfefferle <pe...@japf.ch>
> Sent: Wednesday, June 26, 2019 7:51:43 AM
> To: dev@crail.apache.org; David Crespi
> Subject: Re: Question on crail type-1 with terasort shuffle data
> 
> David,
> 
> 
> Let's make sure we talk about the same thing, I assume with type 1 
>and 2 you
> mean storage classes?
> 
> Regards,
> Jonas
> 
>  On Wed, 26 Jun 2019 14:25:52 +0000
>  David Crespi <da...@storedgesystems.com> wrote:
>> I’m a bit confused now. If it only ever handles 1 namespace, then
>>multiple datanodes are useless.
>>
>> Or are you saying only 1 namespace for type-1?  If you’re not saying
>>only one for type-1, how do you
>>
>> mix types if limited to 1 namespace?
>>
>>
>> Regards,
>>
>>
>>           David
>>
>>
>> ________________________________
>>From: Jonas Pfefferle <pe...@japf.ch>
>> Sent: Wednesday, June 26, 2019 12:25:38 AM
>> To: dev@crail.apache.org; David Crespi
>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>
>> I forgot to add: Namespace has to be 1 for the moment. This is a
>>limitation
>> in how the namenode handles storage tier metadata (essentially it
>>only
>> stores ip/port). You can set it on the server side however it will
>>not be
>> respected by the client, i.e. the client will always use namespace
>>id 1.
>>
>>
>> Regards,
>> Jonas
>>
>>  On Wed, 26 Jun 2019 09:17:40 +0200
>>  "Jonas Pfefferle" <pe...@japf.ch> wrote:
>>> Hi David,
>>>
>>>
>>> This is currently a limitation of the NVMf storage tier. You need to
>>>use the same NQN on all datanodes. The port can be different however.
>>>I will clarify this in the documentation.
>>>
>>> Regards,
>>> Jonas
>>>
>>>  On Tue, 25 Jun 2019 19:43:04 +0000
>>>  David Crespi <da...@storedgesystems.com> wrote:
>>>> It occurred to me that perhaps since I’m using the NVMf, that the
>>>>namenode can only handle one
>>>>
>>>> Subsystem.  And for the type-1, these resources would need to be
>>>>added under the same subsystem,
>>>>
>>>> but as different namespace’s.  Is this correct?  I did try this and
>>>>it appears to be working. If this IS the
>>>>
>>>> correct way of utilizing these, how do I know which namespace(s) are
>>>>being accessed?  I don’t see
>>>>
>>>> anything in the logs that identify which is being addressed.
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>           David
>>>>
>>>>
>>>> ________________________________
>>>>From: David Crespi <da...@storedgesystems.com>
>>>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>>>> To: dev@crail.apache.org
>>>> Subject: Question on crail type-1 with terasort shuffle data
>>>>
>>>> I’m starting to run terasort now with the shuffle data going to
>>>>crail type-1.
>>>> I’ve got the type-2 set to nqn:
>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>>>> which appears to be working okay for the result data.  When I
>>>>introduce type-1 into my config,
>>>> it looks like the namenode gets confused and picks the nqn of the
>>>>type-2 crailstore, instead of
>>>> the one it is assigned (bolded in red below).
>>>>
>>>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
>>>>then 5 type-1 added to the
>>>> spark worker nodes (192.168.3.101-105).  All are running the same
>>>>image as the namenode,
>>>> which is the spark master.
>>>>
>>>> Is there some additional setting I need to include for this?
>>>> These are the variables that get set for each container (added via
>>>>environment)
>>>> # NVMf storage
>>>> crail.storage.nvmf.ip                     $NVMF_IP
>>>> crail.storage.nvmf.port                 $NVMF_PORT
>>>> crail.storage.nvmf.nqn                  $NVMF_NQN
>>>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>>>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>>>>
>>>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>>>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>>>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>>>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>>>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>>>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>>>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>>>> 19/06/25 10:46:43 INFO crail: crail.user crail
>>>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>>>> 19/06/25 10:46:43 INFO crail: crail.debug true
>>>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>>>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>>>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>>>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>>>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>>>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>>>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>>>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>>>org.apache.crail.memory.MappedBufferCache
>>>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>>>crail://192.168.1.164:9060
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>>>roundrobin
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>>>org.apache.crail.namenode.rpc.tcp.TcpNameNode
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>>>org.apache.crail.storage.nvmf.NvmfStorageTier
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>>>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>>>bufferCount 1024
>>>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>>>nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>>>1073741824
>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>>>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>>>queueDepth 32, messageSize 512, nodealy true
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>>>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>>>/192.168.1.164:9060
>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>>>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>streamId 1, isDir true, writeHint 0
>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>address = /192.168.3.100:4420, subsystem NQN =
>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
>>>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>>>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>>>size 28, native size 16
>>>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
>>>>native size 32
>>>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
>>>>72, native size 128
>>>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>>>native size 48
>>>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
>>>>native size 16
>>>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
>>>>40, native size 40
>>>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
>>>>native size 48
>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>139964194703408
>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>/192.168.3.100:4420
>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>461, pd 1
>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>139962246807056
>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
>>>>64
>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>>>send_wr size 32, recv_wr_size 32
>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress
>>>>/192.168.3.100:4420
>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>/192.168.3.100:4420
>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>461, pd 1
>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>139962246807056
>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
>>>>128
>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>>>send_wr size 64, recv_wr_size 64
>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress
>>>>/192.168.3.100:4420
>>>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>>>/192.168.3.100:4420, fsId 0, cache size 1
>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
>>>>success
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>streamId 1, capacity 262656
>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>>>DIRECTORY, storageAffinity 0, locationAffinity 0
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>>streamId 2, isDir true, writeHint 0
>>>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>>>fsId 0
>>>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
>>>>4, token 0
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>>streamId 2, capacity 524800
>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>>>type DIRECTORY, storageAffinity 0, locationAffinity 0
>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>>>fd 4, streamId 3, isDir true, writeHint 0
>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>address = /192.168.3.104:4420, subsystem NQN =
>>>>nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
>>>>nqn.2018-12.com.StorEdgeSystems:worker-4)
>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>139964195131168
>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>/192.168.3.104:4420
>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>461, pd 1
>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>139962246807056
>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
>>>>64
>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>>>send_wr size 32, recv_wr_size 32
>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress
>>>>/192.168.3.104:4420
>>>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>>>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>>>successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
>>>>The host is not allowed to establish an association to any controller
>>>>in the NVM subsystem or the host is not allowed to establish an
>>>>association to the specified controller., CID: 0, Do_not_retry:
>>>>false, More: false, SQHD: 0}
>>>>        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>>>        at
>>>>com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>>>
>>>>
>>>> Regards,
>>>>
>>>>           David
>>>>
>>>

RE: Question on crail type-1 with terasort shuffle data

Posted by David Crespi <da...@storedgesystems.com>.

Yep.  I wanted to use crail-store for temp results storage, storage class 2, AND the shuffle data, with

storage class 1.



Regards,



           David





________________________________
From: Jonas Pfefferle <pe...@japf.ch>
Sent: Wednesday, June 26, 2019 7:51:43 AM
To: dev@crail.apache.org; David Crespi
Subject: Re: Question on crail type-1 with terasort shuffle data

David,


Let's make sure we talk about the same thing, I assume with type 1 and 2 you
mean storage classes?

Regards,
Jonas

  On Wed, 26 Jun 2019 14:25:52 +0000
  David Crespi <da...@storedgesystems.com> wrote:
> I’m a bit confused now. If it only ever handles 1 namespace, then
>multiple datanodes are useless.
>
> Or are you saying only 1 namespace for type-1?  If you’re not saying
>only one for type-1, how do you
>
> mix types if limited to 1 namespace?
>
>
> Regards,
>
>
>           David
>
>
> ________________________________
>From: Jonas Pfefferle <pe...@japf.ch>
> Sent: Wednesday, June 26, 2019 12:25:38 AM
> To: dev@crail.apache.org; David Crespi
> Subject: Re: Question on crail type-1 with terasort shuffle data
>
> I forgot to add: Namespace has to be 1 for the moment. This is a
>limitation
> in how the namenode handles storage tier metadata (essentially it
>only
> stores ip/port). You can set it on the server side however it will
>not be
> respected by the client, i.e. the client will always use namespace
>id 1.
>
>
> Regards,
> Jonas
>
>  On Wed, 26 Jun 2019 09:17:40 +0200
>  "Jonas Pfefferle" <pe...@japf.ch> wrote:
>> Hi David,
>>
>>
>> This is currently a limitation of the NVMf storage tier. You need to
>>use the same NQN on all datanodes. The port can be different however.
>>I will clarify this in the documentation.
>>
>> Regards,
>> Jonas
>>
>>  On Tue, 25 Jun 2019 19:43:04 +0000
>>  David Crespi <da...@storedgesystems.com> wrote:
>>> It occurred to me that perhaps since I’m using the NVMf, that the
>>>namenode can only handle one
>>>
>>> Subsystem.  And for the type-1, these resources would need to be
>>>added under the same subsystem,
>>>
>>> but as different namespace’s.  Is this correct?  I did try this and
>>>it appears to be working. If this IS the
>>>
>>> correct way of utilizing these, how do I know which namespace(s) are
>>>being accessed?  I don’t see
>>>
>>> anything in the logs that identify which is being addressed.
>>>
>>>
>>> Regards,
>>>
>>>
>>>           David
>>>
>>>
>>> ________________________________
>>>From: David Crespi <da...@storedgesystems.com>
>>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>>> To: dev@crail.apache.org
>>> Subject: Question on crail type-1 with terasort shuffle data
>>>
>>> I’m starting to run terasort now with the shuffle data going to
>>>crail type-1.
>>> I’ve got the type-2 set to nqn:
>>>nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>>> which appears to be working okay for the result data.  When I
>>>introduce type-1 into my config,
>>> it looks like the namenode gets confused and picks the nqn of the
>>>type-2 crailstore, instead of
>>> the one it is assigned (bolded in red below).
>>>
>>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
>>>then 5 type-1 added to the
>>> spark worker nodes (192.168.3.101-105).  All are running the same
>>>image as the namenode,
>>> which is the spark master.
>>>
>>> Is there some additional setting I need to include for this?
>>> These are the variables that get set for each container (added via
>>>environment)
>>> # NVMf storage
>>> crail.storage.nvmf.ip                     $NVMF_IP
>>> crail.storage.nvmf.port                 $NVMF_PORT
>>> crail.storage.nvmf.nqn                  $NVMF_NQN
>>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>>>
>>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>>> 19/06/25 10:46:43 INFO crail: crail.user crail
>>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>>> 19/06/25 10:46:43 INFO crail: crail.debug true
>>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>>org.apache.crail.memory.MappedBufferCache
>>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>>crail://192.168.1.164:9060
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>>roundrobin
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>>org.apache.crail.namenode.rpc.tcp.TcpNameNode
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>>org.apache.crail.storage.nvmf.NvmfStorageTier
>>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>>bufferCount 1024
>>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>>nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>>1073741824
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>>queueDepth 32, messageSize 512, nodealy true
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>>/192.168.1.164:9060
>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>streamId 1, isDir true, writeHint 0
>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>address = /192.168.3.100:4420, subsystem NQN =
>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
>>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>>size 28, native size 16
>>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
>>>native size 32
>>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
>>>72, native size 128
>>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>>native size 48
>>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
>>>native size 16
>>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
>>>40, native size 40
>>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
>>>native size 48
>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>139964194703408
>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>/192.168.3.100:4420
>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>461, pd 1
>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>139962246807056
>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
>>>64
>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>>send_wr size 32, recv_wr_size 32
>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress
>>>/192.168.3.100:4420
>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>/192.168.3.100:4420
>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>461, pd 1
>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>139962246807056
>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
>>>128
>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>>send_wr size 64, recv_wr_size 64
>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress
>>>/192.168.3.100:4420
>>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>>/192.168.3.100:4420, fsId 0, cache size 1
>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
>>>success
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>streamId 1, capacity 262656
>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>>DIRECTORY, storageAffinity 0, locationAffinity 0
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>streamId 2, isDir true, writeHint 0
>>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>>fsId 0
>>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
>>>4, token 0
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>streamId 2, capacity 524800
>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>>type DIRECTORY, storageAffinity 0, locationAffinity 0
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>>fd 4, streamId 3, isDir true, writeHint 0
>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>address = /192.168.3.104:4420, subsystem NQN =
>>>nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
>>>nqn.2018-12.com.StorEdgeSystems:worker-4)
>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>139964195131168
>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>/192.168.3.104:4420
>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>461, pd 1
>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>139962246807056
>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
>>>64
>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>>send_wr size 32, recv_wr_size 32
>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress
>>>/192.168.3.104:4420
>>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>>successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
>>>The host is not allowed to establish an association to any controller
>>>in the NVM subsystem or the host is not allowed to establish an
>>>association to the specified controller., CID: 0, Do_not_retry:
>>>false, More: false, SQHD: 0}
>>>        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>>        at
>>>com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>>
>>>
>>> Regards,
>>>
>>>           David
>>>
>>

Re: Question on crail type-1 with terasort shuffle data

Posted by Jonas Pfefferle <pe...@japf.ch>.

David,


Let's make sure we talk about the same thing, I assume with type 1 and 2 you 
mean storage classes?

Regards,
Jonas

  On Wed, 26 Jun 2019 14:25:52 +0000
  David Crespi <da...@storedgesystems.com> wrote:
> I’m a bit confused now. If it only ever handles 1 namespace, then 
>multiple datanodes are useless.
> 
> Or are you saying only 1 namespace for type-1?  If you’re not saying 
>only one for type-1, how do you
> 
> mix types if limited to 1 namespace?
> 
> 
> Regards,
> 
> 
>           David
> 
> 
> ________________________________
>From: Jonas Pfefferle <pe...@japf.ch>
> Sent: Wednesday, June 26, 2019 12:25:38 AM
> To: dev@crail.apache.org; David Crespi
> Subject: Re: Question on crail type-1 with terasort shuffle data
> 
> I forgot to add: Namespace has to be 1 for the moment. This is a 
>limitation
> in how the namenode handles storage tier metadata (essentially it 
>only
> stores ip/port). You can set it on the server side however it will 
>not be
> respected by the client, i.e. the client will always use namespace 
>id 1.
> 
> 
> Regards,
> Jonas
> 
>  On Wed, 26 Jun 2019 09:17:40 +0200
>  "Jonas Pfefferle" <pe...@japf.ch> wrote:
>> Hi David,
>>
>>
>> This is currently a limitation of the NVMf storage tier. You need to
>>use the same NQN on all datanodes. The port can be different however.
>>I will clarify this in the documentation.
>>
>> Regards,
>> Jonas
>>
>>  On Tue, 25 Jun 2019 19:43:04 +0000
>>  David Crespi <da...@storedgesystems.com> wrote:
>>> It occurred to me that perhaps since I’m using the NVMf, that the
>>>namenode can only handle one
>>>
>>> Subsystem.  And for the type-1, these resources would need to be
>>>added under the same subsystem,
>>>
>>> but as different namespace’s.  Is this correct?  I did try this and
>>>it appears to be working. If this IS the
>>>
>>> correct way of utilizing these, how do I know which namespace(s) are
>>>being accessed?  I don’t see
>>>
>>> anything in the logs that identify which is being addressed.
>>>
>>>
>>> Regards,
>>>
>>>
>>>           David
>>>
>>>
>>> ________________________________
>>>From: David Crespi <da...@storedgesystems.com>
>>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>>> To: dev@crail.apache.org
>>> Subject: Question on crail type-1 with terasort shuffle data
>>>
>>> I’m starting to run terasort now with the shuffle data going to
>>>crail type-1.
>>> I’ve got the type-2 set to nqn:
>>>nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>>> which appears to be working okay for the result data.  When I
>>>introduce type-1 into my config,
>>> it looks like the namenode gets confused and picks the nqn of the
>>>type-2 crailstore, instead of
>>> the one it is assigned (bolded in red below).
>>>
>>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
>>>then 5 type-1 added to the
>>> spark worker nodes (192.168.3.101-105).  All are running the same
>>>image as the namenode,
>>> which is the spark master.
>>>
>>> Is there some additional setting I need to include for this?
>>> These are the variables that get set for each container (added via
>>>environment)
>>> # NVMf storage
>>> crail.storage.nvmf.ip                     $NVMF_IP
>>> crail.storage.nvmf.port                 $NVMF_PORT
>>> crail.storage.nvmf.nqn                  $NVMF_NQN
>>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>>>
>>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>>> 19/06/25 10:46:43 INFO crail: crail.user crail
>>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>>> 19/06/25 10:46:43 INFO crail: crail.debug true
>>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>>org.apache.crail.memory.MappedBufferCache
>>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>>crail://192.168.1.164:9060
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>>roundrobin
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>>org.apache.crail.namenode.rpc.tcp.TcpNameNode
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>>org.apache.crail.storage.nvmf.NvmfStorageTier
>>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>>bufferCount 1024
>>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>>nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>>1073741824
>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>>queueDepth 32, messageSize 512, nodealy true
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>>/192.168.1.164:9060
>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>streamId 1, isDir true, writeHint 0
>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>address = /192.168.3.100:4420, subsystem NQN =
>>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
>>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>>size 28, native size 16
>>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
>>>native size 32
>>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
>>>72, native size 128
>>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>>native size 48
>>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
>>>native size 16
>>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
>>>40, native size 40
>>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
>>>native size 48
>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>139964194703408
>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>/192.168.3.100:4420
>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>461, pd 1
>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>139962246807056
>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
>>>64
>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>>send_wr size 32, recv_wr_size 32
>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress
>>>/192.168.3.100:4420
>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>/192.168.3.100:4420
>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>461, pd 1
>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>139962246807056
>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
>>>128
>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>>send_wr size 64, recv_wr_size 64
>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress
>>>/192.168.3.100:4420
>>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>>/192.168.3.100:4420, fsId 0, cache size 1
>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
>>>success
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>streamId 1, capacity 262656
>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>>DIRECTORY, storageAffinity 0, locationAffinity 0
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>>streamId 2, isDir true, writeHint 0
>>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>>fsId 0
>>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
>>>4, token 0
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>>streamId 2, capacity 524800
>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>>type DIRECTORY, storageAffinity 0, locationAffinity 0
>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>>fd 4, streamId 3, isDir true, writeHint 0
>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>address = /192.168.3.104:4420, subsystem NQN =
>>>nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
>>>nqn.2018-12.com.StorEdgeSystems:worker-4)
>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>139964195131168
>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>/192.168.3.104:4420
>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>461, pd 1
>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>139962246807056
>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
>>>64
>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>>send_wr size 32, recv_wr_size 32
>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress
>>>/192.168.3.104:4420
>>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>>successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
>>>The host is not allowed to establish an association to any controller
>>>in the NVM subsystem or the host is not allowed to establish an
>>>association to the specified controller., CID: 0, Do_not_retry:
>>>false, More: false, SQHD: 0}
>>>        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>>        at
>>>com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>>
>>>
>>> Regards,
>>>
>>>           David
>>>
>>

RE: Question on crail type-1 with terasort shuffle data

Posted by David Crespi <da...@storedgesystems.com>.

I’m a bit confused now. If it only ever handles 1 namespace, then multiple datanodes are useless.

Or are you saying only 1 namespace for type-1?  If you’re not saying only one for type-1, how do you

mix types if limited to 1 namespace?



Regards,



           David





________________________________
From: Jonas Pfefferle <pe...@japf.ch>
Sent: Wednesday, June 26, 2019 12:25:38 AM
To: dev@crail.apache.org; David Crespi
Subject: Re: Question on crail type-1 with terasort shuffle data

I forgot to add: Namespace has to be 1 for the moment. This is a limitation
in how the namenode handles storage tier metadata (essentially it only
stores ip/port). You can set it on the server side however it will not be
respected by the client, i.e. the client will always use namespace id 1.


Regards,
Jonas

  On Wed, 26 Jun 2019 09:17:40 +0200
  "Jonas Pfefferle" <pe...@japf.ch> wrote:
> Hi David,
>
>
> This is currently a limitation of the NVMf storage tier. You need to
>use the same NQN on all datanodes. The port can be different however.
>I will clarify this in the documentation.
>
> Regards,
> Jonas
>
>  On Tue, 25 Jun 2019 19:43:04 +0000
>  David Crespi <da...@storedgesystems.com> wrote:
>> It occurred to me that perhaps since I’m using the NVMf, that the
>>namenode can only handle one
>>
>> Subsystem.  And for the type-1, these resources would need to be
>>added under the same subsystem,
>>
>> but as different namespace’s.  Is this correct?  I did try this and
>>it appears to be working. If this IS the
>>
>> correct way of utilizing these, how do I know which namespace(s) are
>>being accessed?  I don’t see
>>
>> anything in the logs that identify which is being addressed.
>>
>>
>> Regards,
>>
>>
>>           David
>>
>>
>> ________________________________
>>From: David Crespi <da...@storedgesystems.com>
>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>> To: dev@crail.apache.org
>> Subject: Question on crail type-1 with terasort shuffle data
>>
>> I’m starting to run terasort now with the shuffle data going to
>>crail type-1.
>> I’ve got the type-2 set to nqn:
>>nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>> which appears to be working okay for the result data.  When I
>>introduce type-1 into my config,
>> it looks like the namenode gets confused and picks the nqn of the
>>type-2 crailstore, instead of
>> the one it is assigned (bolded in red below).
>>
>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and
>>then 5 type-1 added to the
>> spark worker nodes (192.168.3.101-105).  All are running the same
>>image as the namenode,
>> which is the spark master.
>>
>> Is there some additional setting I need to include for this?
>> These are the variables that get set for each container (added via
>>environment)
>> # NVMf storage
>> crail.storage.nvmf.ip                     $NVMF_IP
>> crail.storage.nvmf.port                 $NVMF_PORT
>> crail.storage.nvmf.nqn                  $NVMF_NQN
>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>>
>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>> 19/06/25 10:46:43 INFO crail: crail.user crail
>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>> 19/06/25 10:46:43 INFO crail: crail.debug true
>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>org.apache.crail.memory.MappedBufferCache
>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>crail://192.168.1.164:9060
>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>roundrobin
>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>org.apache.crail.namenode.rpc.tcp.TcpNameNode
>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>org.apache.crail.storage.nvmf.NvmfStorageTier
>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>bufferCount 1024
>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>1073741824
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>queueDepth 32, messageSize 512, nodealy true
>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>/192.168.1.164:9060
>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>streamId 1, isDir true, writeHint 0
>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>address = /192.168.3.100:4420, subsystem NQN =
>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>size 28, native size 16
>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32,
>>native size 32
>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size
>>72, native size 128
>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>native size 48
>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16,
>>native size 16
>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size
>>40, native size 40
>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48,
>>native size 48
>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>139964194703408
>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>461, pd 1
>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>139962246807056
>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe
>>64
>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>send_wr size 32, recv_wr_size 32
>> 19/06/25 10:46:43 INFO disni: connect, id 0
>> 19/06/25 10:46:43 INFO disni: got event type +
>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>461, pd 1
>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>139962246807056
>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe
>>128
>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>send_wr size 64, recv_wr_size 64
>> 19/06/25 10:46:43 INFO disni: connect, id 0
>> 19/06/25 10:46:43 INFO disni: got event type +
>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>/192.168.3.100:4420, fsId 0, cache size 1
>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true,
>>success
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>streamId 1, capacity 262656
>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>DIRECTORY, storageAffinity 0, locationAffinity 0
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0,
>>streamId 2, isDir true, writeHint 0
>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>fsId 0
>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd
>>4, token 0
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0,
>>streamId 2, capacity 524800
>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>type DIRECTORY, storageAffinity 0, locationAffinity 0
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>fd 4, streamId 3, isDir true, writeHint 0
>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>address = /192.168.3.104:4420, subsystem NQN =
>>nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
>>nqn.2018-12.com.StorEdgeSystems:worker-4)
>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>139964195131168
>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>/192.168.3.104:4420
>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>461, pd 1
>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>139962246807056
>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe
>>64
>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>send_wr size 32, recv_wr_size 32
>> 19/06/25 10:46:43 INFO disni: connect, id 0
>> 19/06/25 10:46:43 INFO disni: got event type +
>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress
>>/192.168.3.104:4420
>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 -
>>The host is not allowed to establish an association to any controller
>>in the NVM subsystem or the host is not allowed to establish an
>>association to the specified controller., CID: 0, Do_not_retry:
>>false, More: false, SQHD: 0}
>>        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>        at
>>com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>
>>
>> Regards,
>>
>>           David
>>
>

Re: Question on crail type-1 with terasort shuffle data

Posted by Jonas Pfefferle <pe...@japf.ch>.

I forgot to add: Namespace has to be 1 for the moment. This is a limitation 
in how the namenode handles storage tier metadata (essentially it only 
stores ip/port). You can set it on the server side however it will not be 
respected by the client, i.e. the client will always use namespace id 1.


Regards,
Jonas

  On Wed, 26 Jun 2019 09:17:40 +0200
  "Jonas Pfefferle" <pe...@japf.ch> wrote:
> Hi David,
> 
> 
> This is currently a limitation of the NVMf storage tier. You need to 
>use the same NQN on all datanodes. The port can be different however. 
>I will clarify this in the documentation.
> 
> Regards,
> Jonas
> 
>  On Tue, 25 Jun 2019 19:43:04 +0000
>  David Crespi <da...@storedgesystems.com> wrote:
>> It occurred to me that perhaps since I’m using the NVMf, that the 
>>namenode can only handle one
>> 
>> Subsystem.  And for the type-1, these resources would need to be 
>>added under the same subsystem,
>> 
>> but as different namespace’s.  Is this correct?  I did try this and 
>>it appears to be working. If this IS the
>> 
>> correct way of utilizing these, how do I know which namespace(s) are 
>>being accessed?  I don’t see
>> 
>> anything in the logs that identify which is being addressed.
>> 
>> 
>> Regards,
>> 
>> 
>>           David
>> 
>> 
>> ________________________________
>>From: David Crespi <da...@storedgesystems.com>
>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>> To: dev@crail.apache.org
>> Subject: Question on crail type-1 with terasort shuffle data
>> 
>> I’m starting to run terasort now with the shuffle data going to 
>>crail type-1.
>> I’ve got the type-2 set to nqn: 
>>nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>> which appears to be working okay for the result data.  When I 
>>introduce type-1 into my config,
>> it looks like the namenode gets confused and picks the nqn of the 
>>type-2 crailstore, instead of
>> the one it is assigned (bolded in red below).
>> 
>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and 
>>then 5 type-1 added to the
>> spark worker nodes (192.168.3.101-105).  All are running the same 
>>image as the namenode,
>> which is the spark master.
>> 
>> Is there some additional setting I need to include for this?
>> These are the variables that get set for each container (added via 
>>environment)
>> # NVMf storage
>> crail.storage.nvmf.ip                     $NVMF_IP
>> crail.storage.nvmf.port                 $NVMF_PORT
>> crail.storage.nvmf.nqn                  $NVMF_NQN
>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>> 
>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>> 19/06/25 10:46:43 INFO crail: crail.user crail
>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>> 19/06/25 10:46:43 INFO crail: crail.debug true
>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl 
>>org.apache.crail.memory.MappedBufferCache
>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>> 19/06/25 10:46:43 INFO crail: crail.namenode.address 
>>crail://192.168.1.164:9060
>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection 
>>roundrobin
>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype 
>>org.apache.crail.namenode.rpc.tcp.TcpNameNode
>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>> 19/06/25 10:46:43 INFO crail: crail.storage.types 
>>org.apache.crail.storage.nvmf.NvmfStorageTier
>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0, 
>>bufferCount 1024
>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn 
>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn 
>>nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize 
>>1073741824
>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0, 
>>queueDepth 32, messageSize 512, nodealy true
>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>> 19/06/25 10:46:43 INFO crail: connected to namenode(s) 
>>/192.168.1.164:9060
>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, 
>>streamId 1, isDir true, writeHint 0
>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport 
>>address = /192.168.3.100:4420, subsystem NQN = 
>>nqn.2018-12.com.StorEdgeSystems:cntlr13
>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs 
>>size 28, native size 16
>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32, 
>>native size 32
>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size 
>>72, native size 128
>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48, 
>>native size 48
>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16, 
>>native size 16
>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size 
>>40, native size 40
>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48, 
>>native size 48
>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId 
>>139964194703408
>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres 
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context 
>>461, pd 1
>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>> 19/06/25 10:46:43 INFO disni: createCompChannel, context 
>>139962246807056
>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe 
>>64
>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176, 
>>send_wr size 32, recv_wr_size 32
>> 19/06/25 10:46:43 INFO disni: connect, id 0
>> 19/06/25 10:46:43 INFO disni: got event type + 
>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress 
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres 
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context 
>>461, pd 1
>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>> 19/06/25 10:46:43 INFO disni: createCompChannel, context 
>>139962246807056
>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe 
>>128
>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304, 
>>send_wr size 64, recv_wr_size 64
>> 19/06/25 10:46:43 INFO disni: connect, id 0
>> 19/06/25 10:46:43 INFO disni: got event type + 
>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress 
>>/192.168.3.100:4420
>> 19/06/25 10:46:43 INFO crail: EndpointCache miss 
>>/192.168.3.100:4420, fsId 0, cache size 1
>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true, 
>>success
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, 
>>streamId 1, capacity 262656
>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type 
>>DIRECTORY, storageAffinity 0, locationAffinity 0
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, 
>>streamId 2, isDir true, writeHint 0
>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420, 
>>fsId 0
>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd 
>>4, token 0
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, 
>>streamId 2, capacity 524800
>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast, 
>>type DIRECTORY, storageAffinity 0, locationAffinity 0
>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark, 
>>fd 4, streamId 3, isDir true, writeHint 0
>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport 
>>address = /192.168.3.104:4420, subsystem NQN = 
>>nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be 
>>nqn.2018-12.com.StorEdgeSystems:worker-4)
>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId 
>>139964195131168
>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres 
>>/192.168.3.104:4420
>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context 
>>461, pd 1
>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>> 19/06/25 10:46:43 INFO disni: createCompChannel, context 
>>139962246807056
>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe 
>>64
>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896, 
>>send_wr size 32, recv_wr_size 32
>> 19/06/25 10:46:43 INFO disni: connect, id 0
>> 19/06/25 10:46:43 INFO disni: got event type + 
>>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress 
>>/192.168.3.104:4420
>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not 
>>successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 - 
>>The host is not allowed to establish an association to any controller 
>>in the NVM subsystem or the host is not allowed to establish an 
>>association to the specified controller., CID: 0, Do_not_retry: 
>>false, More: false, SQHD: 0}
>>        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>        at 
>>com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>> 
>> 
>> Regards,
>> 
>>           David
>> 
>

Re: Question on crail type-1 with terasort shuffle data

Posted by Jonas Pfefferle <pe...@japf.ch>.

Hi David,


This is currently a limitation of the NVMf storage tier. You need to use the 
same NQN on all datanodes. The port can be different however. I will clarify 
this in the documentation.

Regards,
Jonas

  On Tue, 25 Jun 2019 19:43:04 +0000
  David Crespi <da...@storedgesystems.com> wrote:
> It occurred to me that perhaps since I’m using the NVMf, that the 
>namenode can only handle one
> 
> Subsystem.  And for the type-1, these resources would need to be 
>added under the same subsystem,
> 
> but as different namespace’s.  Is this correct?  I did try this and 
>it appears to be working. If this IS the
> 
> correct way of utilizing these, how do I know which namespace(s) are 
>being accessed?  I don’t see
> 
> anything in the logs that identify which is being addressed.
> 
> 
> Regards,
> 
> 
>           David
> 
> 
> ________________________________
>From: David Crespi <da...@storedgesystems.com>
> Sent: Tuesday, June 25, 2019 11:09:56 AM
> To: dev@crail.apache.org
> Subject: Question on crail type-1 with terasort shuffle data
> 
> I’m starting to run terasort now with the shuffle data going to 
>crail type-1.
> I’ve got the type-2 set to nqn: 
>nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
> which appears to be working okay for the result data.  When I 
>introduce type-1 into my config,
> it looks like the namenode gets confused and picks the nqn of the 
>type-2 crailstore, instead of
> the one it is assigned (bolded in red below).
> 
> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and 
>then 5 type-1 added to the
> spark worker nodes (192.168.3.101-105).  All are running the same 
>image as the namenode,
> which is the spark master.
> 
> Is there some additional setting I need to include for this?
> These are the variables that get set for each container (added via 
>environment)
> # NVMf storage
> crail.storage.nvmf.ip                     $NVMF_IP
> crail.storage.nvmf.port                 $NVMF_PORT
> crail.storage.nvmf.nqn                  $NVMF_NQN
> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
> 
> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
> 19/06/25 10:46:43 INFO crail: crail.version 3101
> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
> 19/06/25 10:46:43 INFO crail: crail.user crail
> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
> 19/06/25 10:46:43 INFO crail: crail.debug true
> 19/06/25 10:46:43 INFO crail: crail.statistics true
> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
> 19/06/25 10:46:43 INFO crail: crail.singleton true
> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
> 19/06/25 10:46:43 INFO crail: crail.cacheimpl 
>org.apache.crail.memory.MappedBufferCache
> 19/06/25 10:46:43 INFO crail: crail.locationmap
> 19/06/25 10:46:43 INFO crail: crail.namenode.address 
>crail://192.168.1.164:9060
> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection 
>roundrobin
> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype 
>org.apache.crail.namenode.rpc.tcp.TcpNameNode
> 19/06/25 10:46:43 INFO crail: crail.namenode.log
> 19/06/25 10:46:43 INFO crail: crail.storage.types 
>org.apache.crail.storage.nvmf.NvmfStorageTier
> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0, 
>bufferCount 1024
> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn 
>nqn.2018-12.com.StorEdgeSystems:cntlr13
> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn 
>nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize 
>1073741824
> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0, 
>queueDepth 32, messageSize 512, nodealy true
> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
> 19/06/25 10:46:43 INFO crail: connected to namenode(s) 
>/192.168.1.164:9060
> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, 
>streamId 1, isDir true, writeHint 0
> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport 
>address = /192.168.3.100:4420, subsystem NQN = 
>nqn.2018-12.com.StorEdgeSystems:cntlr13
> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs 
>size 28, native size 16
> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32, 
>native size 32
> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size 
>72, native size 128
> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48, 
>native size 48
> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16, 
>native size 16
> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size 
>40, native size 40
> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48, 
>native size 48
> 19/06/25 10:46:43 INFO disni: createEventChannel, objId 
>139964194703408
> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
> 19/06/25 10:46:43 INFO disni: resolveAddr, addres 
>/192.168.3.100:4420
> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
> 19/06/25 10:46:43 INFO disni: setting up protection domain, context 
>461, pd 1
> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
> 19/06/25 10:46:43 INFO disni: createCompChannel, context 
>139962246807056
> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe 
>64
> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176, 
>send_wr size 32, recv_wr_size 32
> 19/06/25 10:46:43 INFO disni: connect, id 0
> 19/06/25 10:46:43 INFO disni: got event type + 
>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress 
>/192.168.3.100:4420
> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
> 19/06/25 10:46:43 INFO disni: resolveAddr, addres 
>/192.168.3.100:4420
> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
> 19/06/25 10:46:43 INFO disni: setting up protection domain, context 
>461, pd 1
> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
> 19/06/25 10:46:43 INFO disni: createCompChannel, context 
>139962246807056
> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe 
>128
> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304, 
>send_wr size 64, recv_wr_size 64
> 19/06/25 10:46:43 INFO disni: connect, id 0
> 19/06/25 10:46:43 INFO disni: got event type + 
>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress 
>/192.168.3.100:4420
> 19/06/25 10:46:43 INFO crail: EndpointCache miss 
>/192.168.3.100:4420, fsId 0, cache size 1
> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true, 
>success
> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, 
>streamId 1, capacity 262656
> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type 
>DIRECTORY, storageAffinity 0, locationAffinity 0
> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, 
>streamId 2, isDir true, writeHint 0
> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420, 
>fsId 0
> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd 
>4, token 0
> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, 
>streamId 2, capacity 524800
> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast, 
>type DIRECTORY, storageAffinity 0, locationAffinity 0
> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark, 
>fd 4, streamId 3, isDir true, writeHint 0
> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport 
>address = /192.168.3.104:4420, subsystem NQN = 
>nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be 
>nqn.2018-12.com.StorEdgeSystems:worker-4)
> 19/06/25 10:46:43 INFO disni: createEventChannel, objId 
>139964195131168
> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
> 19/06/25 10:46:43 INFO disni: resolveAddr, addres 
>/192.168.3.104:4420
> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
> 19/06/25 10:46:43 INFO disni: setting up protection domain, context 
>461, pd 1
> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
> 19/06/25 10:46:43 INFO disni: createCompChannel, context 
>139962246807056
> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe 
>64
> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896, 
>send_wr size 32, recv_wr_size 32
> 19/06/25 10:46:43 INFO disni: connect, id 0
> 19/06/25 10:46:43 INFO disni: got event type + 
>RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress 
>/192.168.3.104:4420
> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
> com.ibm.jnvmf.UnsuccessfulComandException: Command was not 
>successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 - 
>The host is not allowed to establish an association to any controller 
>in the NVM subsystem or the host is not allowed to establish an 
>association to the specified controller., CID: 0, Do_not_retry: 
>false, More: false, SQHD: 0}
>        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>        at 
>com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
> 
> 
> Regards,
> 
>           David
>

RE: Question on crail type-1 with terasort shuffle data

Posted by David Crespi <da...@storedgesystems.com>.

It occurred to me that perhaps since I’m using the NVMf, that the namenode can only handle one

Subsystem.  And for the type-1, these resources would need to be added under the same subsystem,

but as different namespace’s.  Is this correct?  I did try this and it appears to be working. If this IS the

correct way of utilizing these, how do I know which namespace(s) are being accessed?  I don’t see

anything in the logs that identify which is being addressed.



Regards,



           David





________________________________
From: David Crespi <da...@storedgesystems.com>
Sent: Tuesday, June 25, 2019 11:09:56 AM
To: dev@crail.apache.org
Subject: Question on crail type-1 with terasort shuffle data

I’m starting to run terasort now with the shuffle data going to crail type-1.
I’ve got the type-2 set to nqn: nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
which appears to be working okay for the result data.  When I introduce type-1 into my config,
it looks like the namenode gets confused and picks the nqn of the type-2 crailstore, instead of
the one it is assigned (bolded in red below).

In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5) and then 5 type-1 added to the
spark worker nodes (192.168.3.101-105).  All are running the same image as the namenode,
which is the spark master.

Is there some additional setting I need to include for this?
These are the variables that get set for each container (added via environment)
# NVMf storage
crail.storage.nvmf.ip                     $NVMF_IP
crail.storage.nvmf.port                 $NVMF_PORT
crail.storage.nvmf.nqn                  $NVMF_NQN
crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
crail.storage.nvmf.namespace      $NVMF_NAMESPACE

19/06/25 10:46:43 INFO crail: creating singleton crail file system
19/06/25 10:46:43 INFO crail: crail.version 3101
19/06/25 10:46:43 INFO crail: crail.directorydepth 16
19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
19/06/25 10:46:43 INFO crail: crail.cachelimit 0
19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
19/06/25 10:46:43 INFO crail: crail.user crail
19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
19/06/25 10:46:43 INFO crail: crail.debug true
19/06/25 10:46:43 INFO crail: crail.statistics true
19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
19/06/25 10:46:43 INFO crail: crail.slicesize 65536
19/06/25 10:46:43 INFO crail: crail.singleton true
19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
19/06/25 10:46:43 INFO crail: crail.cacheimpl org.apache.crail.memory.MappedBufferCache
19/06/25 10:46:43 INFO crail: crail.locationmap
19/06/25 10:46:43 INFO crail: crail.namenode.address crail://192.168.1.164:9060
19/06/25 10:46:43 INFO crail: crail.namenode.blockselection roundrobin
19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
19/06/25 10:46:43 INFO crail: crail.namenode.rpctype org.apache.crail.namenode.rpc.tcp.TcpNameNode
19/06/25 10:46:43 INFO crail: crail.namenode.log
19/06/25 10:46:43 INFO crail: crail.storage.types org.apache.crail.storage.nvmf.NvmfStorageTier
19/06/25 10:46:43 INFO crail: crail.storage.classes 2
19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0, bufferCount 1024
19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn nqn.2018-12.com.StorEdgeSystems:cntlr13
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize 1073741824
19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0, queueDepth 32, messageSize 512, nodealy true
19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize 512
19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
19/06/25 10:46:43 INFO crail: connected to namenode(s) /192.168.1.164:9060
19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd 2
19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true
19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, streamId 1, isDir true, writeHint 0
19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport address = /192.168.3.100:4420, subsystem NQN = nqn.2018-12.com.StorEdgeSystems:cntlr13
19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type 'nat'
19/06/25 10:46:43 INFO disni: jverbs jni version 32
19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs size 28, native size 16
19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size 32, native size 32
19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs size 72, native size 128
19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48, native size 48
19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size 16, native size 16
19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs size 40, native size 40
19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size 48, native size 48
19/06/25 10:46:43 INFO disni: createEventChannel, objId 139964194703408
19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
19/06/25 10:46:43 INFO disni: createId, id 139964194793600
19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv 0
19/06/25 10:46:43 INFO disni: resolveAddr, addres /192.168.3.100:4420
19/06/25 10:46:43 INFO disni: resolveRoute, id 0
19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
19/06/25 10:46:43 INFO disni: setting up protection domain, context 461, pd 1
19/06/25 10:46:43 INFO disni: new endpoint CQ processor
19/06/25 10:46:43 INFO disni: createCompChannel, context 139962246807056
19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488, ncqe 64
19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176, send_wr size 32, recv_wr_size 32
19/06/25 10:46:43 INFO disni: connect, id 0
19/06/25 10:46:43 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, dstAddress /192.168.3.100:4420
19/06/25 10:46:43 INFO disni: createId, id 139964195036000
19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv 0
19/06/25 10:46:43 INFO disni: resolveAddr, addres /192.168.3.100:4420
19/06/25 10:46:43 INFO disni: resolveRoute, id 0
19/06/25 10:46:43 INFO disni: setting up protection domain, context 461, pd 1
19/06/25 10:46:43 INFO disni: new endpoint CQ processor
19/06/25 10:46:43 INFO disni: createCompChannel, context 139962246807056
19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752, ncqe 128
19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304, send_wr size 64, recv_wr_size 64
19/06/25 10:46:43 INFO disni: connect, id 0
19/06/25 10:46:43 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, dstAddress /192.168.3.100:4420
19/06/25 10:46:43 INFO crail: EndpointCache miss /192.168.3.100:4420, fsId 0, cache size 1
19/06/25 10:46:43 INFO crail: delete: name /spark, recursive true, success
19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, streamId 1, capacity 262656
19/06/25 10:46:43 INFO crail: createNode: name /spark, type DIRECTORY, storageAffinity 0, locationAffinity 0
19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /, fd 0, streamId 2, isDir true, writeHint 0
19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420, fsId 0
19/06/25 10:46:43 INFO crail: createFile: name /spark, success, fd 4, token 0
19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /, fd 0, streamId 2, capacity 524800
19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast, type DIRECTORY, storageAffinity 0, locationAffinity 0
19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark, fd 4, streamId 3, isDir true, writeHint 0
19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport address = /192.168.3.104:4420, subsystem NQN = nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be nqn.2018-12.com.StorEdgeSystems:worker-4)
19/06/25 10:46:43 INFO disni: createEventChannel, objId 139964195131168
19/06/25 10:46:43 INFO disni: createId, id 139964195141520
19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv 0
19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel 0
19/06/25 10:46:43 INFO disni: resolveAddr, addres /192.168.3.104:4420
19/06/25 10:46:43 INFO disni: resolveRoute, id 0
19/06/25 10:46:43 INFO disni: setting up protection domain, context 461, pd 1
19/06/25 10:46:43 INFO disni: new endpoint CQ processor
19/06/25 10:46:43 INFO disni: createCompChannel, context 139962246807056
19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224, ncqe 64
19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896, send_wr size 32, recv_wr_size 32
19/06/25 10:46:43 INFO disni: connect, id 0
19/06/25 10:46:43 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, dstAddress /192.168.3.104:4420
19/06/25 10:46:43 INFO crail: ERROR: failed data operation
com.ibm.jnvmf.UnsuccessfulComandException: Command was not successful. {StatusCodeType: 1 - Command Specific, SatusCode: 132 - The host is not allowed to establish an association to any controller in the NVM subsystem or the host is not allowed to establish an association to the specified controller., CID: 0, Do_not_retry: false, More: false, SQHD: 0}
        at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
        at com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)


Regards,

           David