[GitHub] leleamol commented on a change in pull request #9367: Limit the test_nccl to run on 8 GPUs only until NCCL2.1 issue is fixed.

[GitBox <gi...@apache.org>]

leleamol commented on a change in pull request #9367: Limit the test_nccl to run on 8 GPUs only until NCCL2.1 issue is fixed.
URL: https://github.com/apache/incubator-mxnet/pull/9367#discussion_r161350269
 
 

 ##########
 File path: tests/python/gpu/test_nccl.py
 ##########
 @@ -21,7 +21,16 @@
 
 shapes = [(10), (100), (1000), (10000), (100000), (2,2), (2,3,4,5,6,7,8)]
 keys = [1,2,3,4,5,6,7]
-gpus = range(1,1+len(mx.test_utils.list_gpus()))
+num_gpus = len(mx.test_utils.list_gpus())
+
+#TODO The test is capped to run with 8 GPUs only. This is due to a bug in NCCL 2.1
+#It is addressed in github issue https://github.com/apache/incubator-mxnet/issues/9004
+#Remove this constraint when the github issue is fixed.
+
+if num_gpus > 8 :
+    num_gpus = 8;
+
+gpus = range(1,1+num_gpus)
 
 Review comment:
   Thanks @marcoabreu  for the review! I have added the print statement. The output of the test is given below.
   Regarding machine with 2 gpus, the test caps to 8 gpus only if the machine has more than 8 gpus. The test should pass on machine with 2 gpus
   
   Test output:
   
   gpu]$ python test_nccl.py 
   The machine has 16 gpus. We will run the test on 8 gpus only due to a bug in NCCL 2.1.
   Please remove this limitation when issue #9004 is fixed.
   Passed
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services