[GitHub] ptrendx commented on issue #9367: Limit the test_nccl to run on 8 GPUs only until NCCL2.1 issue is fixed.

[GitBox <gi...@apache.org>]

ptrendx commented on issue #9367: Limit the test_nccl to run on 8 GPUs only until NCCL2.1 issue is fixed.
URL: https://github.com/apache/incubator-mxnet/pull/9367#issuecomment-357388871
 
 
   Hi, sorry for replying late.
   
   So there is a little bit of misunderstanding there it seems. The fix to the bug in NCCL 2.1 you are referring to will not make this test work in p2 instance with 16 GPUs (even though the actual NCCL integration works well in that scenario). It should only make the test fail gracefully instead of segfaulting.
   
   There is a limit to the number of P2P peers any given GPU may have (it is a hardware limit for all PCI-E hardware, not just GPUs) - it is 8. NCCL under normal circumstances works fine with p2 instances with 16 GPUs under the same PCI-E tree because it uses a ring for communication, so each GPU establishes P2P access only between its neighbors in a ring.
   
   So why is the test broken then? Because (being a test) it does a sweep over multiple configurations (in this case ring sizes), going from 1 to N (= number of GPUs in a node). During this sweep it produces rings as follows:
   0
   0 - 1 (gpu 0 opens P2P connection to GPU 1)
   0 - 1 - 2 (gpu 0 opens P2P connection to GPU 1 and 2 - it is a ring)
   0 - 1 - 2 - 3 (gpu 0 opens P2P connection to GPU 1 and 3)
   ...
   As you can see, during the execution of the for loop over test cases, GPU 0 opens P2P connections to GPUs 1,2,3,... and at some point reaches its limit of 8 peers (during ring 1 - .... - 9) and then fails during the 1 - ... - 10 ring test case. This is expected behavior (unfortunately). 
   
   To fully fix this test (which you can enable right now, btw, you don't need to wait for the next version of NCCL to do that) you would need to either:
    - split the test into 2 very similar tests - 1 covering range from 1 to 8, and second from 9 to 16 (which is very simple, just copy the test_nccl.py file and in the second copy remove limiter on `num_gpus` and do `gpus=range(9, 1 + num_gpus)` instead of `gpus=range(1, 1 + num_gpus)`)
    - make the test choose gpus assignment to ring smarter, like this (with 16 gpus):
   0
   0 - 1
   15 - 0 - 1
   15 - 0 - 1 - 2
   14 - 15 - 0 - 1 - 2
   ...
   which would ensure that each gpu opens only up to 4 p2p connections.
   
   Does this clarify the confusion? @leleamol 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services