You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@singa.apache.org by GitBox <gi...@apache.org> on 2020/04/06 02:42:32 UTC

[GitHub] [singa-doc] chrishkchris opened a new pull request #16: Add more details in the explanation of dist-train.md

chrishkchris opened a new pull request #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16
 
 
   Add more details in the explanation of dist-train.md

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] chrishkchris commented on a change in pull request #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#discussion_r403812104
 
 

 ##########
 File path: docs-site/docs/dist-train.md
 ##########
 @@ -145,30 +165,80 @@ if __name__ == '__main__':
     nccl_id = singa.NcclIdHolder()
 
     # Define the number of GPUs to be used in the training process
-    gpu_per_node = int(sys.argv[1])
-    gpu_num = 1
+    num_gpus = int(sys.argv[1])
 
     # Define and launch the multi-processing
 	import multiprocessing
     process = []
-    for gpu_num in range(0, gpu_per_node):
+    for gpu_num in range(0, num_gpus):
         process.append(multiprocessing.Process(target=train_mnist_cnn,
-                       args=(nccl_id, gpu_num, gpu_per_node)))
+                       args=(nccl_id, gpu_num, num_gpus)))
 
     for p in process:
         p.start()
 ```
 
+Here are some explanations concerning the variables created above:
+
+(i) `nccl_id`
+
+Note that we need to generate a NCCL ID here to be used for collective communication, and then pass it to all the processes. 
+The NCCL ID is like a ticket, where only the processes with this ID can join the AllReduce operation. 
+(Later if we use MPI, the passing of NCCL ID is not necessary, because the ID is broadcased by MPI in our code automatically)
+
+(ii) `num_gpus`
 
 Review comment:
   yes, can use <local_rank, global_rank, world size> in every locations of python and C++? num_gpus will be changed to world size

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] chrishkchris commented on a change in pull request #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#discussion_r403812369
 
 

 ##########
 File path: docs-site/docs/dist-train.md
 ##########
 @@ -75,12 +75,30 @@ sgd = opt.DistOpt(sgd)
 dev = device.create_cuda_gpu_on(sgd.rank_in_local)
 ```
 
-`dev` represents the `Device` instance, where to load data and run the CNN
+Here are some explanations concerning some variables in the code:
+
+(i) `dev` 
+
+dev represents the `Device` instance, where to load data and run the CNN
 model.
 
+(ii)`rank_in_local` 
 
 Review comment:
   yes, we use <local_rank, global_rank, world size> everywhere

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] nudles merged pull request #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
nudles merged pull request #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] nudles commented on a change in pull request #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#discussion_r403811535
 
 

 ##########
 File path: docs-site/docs/dist-train.md
 ##########
 @@ -20,9 +20,9 @@ the gradients of the weights, which are averaged via All-Reduce (provided by
 [NCCL](https://developer.nvidia.com/nccl)) for weight update following
 stochastic gradient descent algorithms (SGD).
 
-The all-reduce operation by NCCL can be used to reduce and synchronize the
+The All-reduce operation by NCCL can be used to reduce and synchronize the
 
 Review comment:
   All-reduce, All-Reduce, all-reduce?
   just be consistent.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] chrishkchris commented on a change in pull request #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#discussion_r403812437
 
 

 ##########
 File path: docs-site/docs/dist-train.md
 ##########
 @@ -20,9 +20,9 @@ the gradients of the weights, which are averaged via All-Reduce (provided by
 [NCCL](https://developer.nvidia.com/nccl)) for weight update following
 stochastic gradient descent algorithms (SGD).
 
-The all-reduce operation by NCCL can be used to reduce and synchronize the
+The All-reduce operation by NCCL can be used to reduce and synchronize the
 
 Review comment:
   ok, I will change

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] nudles commented on a change in pull request #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#discussion_r403811468
 
 

 ##########
 File path: docs-site/docs/dist-train.md
 ##########
 @@ -75,12 +75,30 @@ sgd = opt.DistOpt(sgd)
 dev = device.create_cuda_gpu_on(sgd.rank_in_local)
 ```
 
-`dev` represents the `Device` instance, where to load data and run the CNN
+Here are some explanations concerning some variables in the code:
+
+(i) `dev` 
+
+dev represents the `Device` instance, where to load data and run the CNN
 model.
 
+(ii)`rank_in_local` 
 
 Review comment:
   local_rank
   global_rank

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] chrishkchris commented on issue #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#issuecomment-609586710
 
 
   I have already unified terms to <local_rank, global_rank, world_size>

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] chrishkchris edited a comment on issue #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#issuecomment-609586710
 
 
   I have already unified terms to <local_rank, global_rank, world_size> in singa code

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] chrishkchris commented on issue #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#issuecomment-609564024
 
 
   I unified the terms in the doc, will change the corresponding python and C communicator code

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] chrishkchris commented on a change in pull request #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#discussion_r403812104
 
 

 ##########
 File path: docs-site/docs/dist-train.md
 ##########
 @@ -145,30 +165,80 @@ if __name__ == '__main__':
     nccl_id = singa.NcclIdHolder()
 
     # Define the number of GPUs to be used in the training process
-    gpu_per_node = int(sys.argv[1])
-    gpu_num = 1
+    num_gpus = int(sys.argv[1])
 
     # Define and launch the multi-processing
 	import multiprocessing
     process = []
-    for gpu_num in range(0, gpu_per_node):
+    for gpu_num in range(0, num_gpus):
         process.append(multiprocessing.Process(target=train_mnist_cnn,
-                       args=(nccl_id, gpu_num, gpu_per_node)))
+                       args=(nccl_id, gpu_num, num_gpus)))
 
     for p in process:
         p.start()
 ```
 
+Here are some explanations concerning the variables created above:
+
+(i) `nccl_id`
+
+Note that we need to generate a NCCL ID here to be used for collective communication, and then pass it to all the processes. 
+The NCCL ID is like a ticket, where only the processes with this ID can join the AllReduce operation. 
+(Later if we use MPI, the passing of NCCL ID is not necessary, because the ID is broadcased by MPI in our code automatically)
+
+(ii) `num_gpus`
 
 Review comment:
   yes, can use <local_rank, global_rank, world size> in every locations of python and C++?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [singa-doc] nudles commented on a change in pull request #16: Add more details in the explanation of dist-train.md

Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #16: Add more details in the explanation of dist-train.md
URL: https://github.com/apache/singa-doc/pull/16#discussion_r403811276
 
 

 ##########
 File path: docs-site/docs/dist-train.md
 ##########
 @@ -145,30 +165,80 @@ if __name__ == '__main__':
     nccl_id = singa.NcclIdHolder()
 
     # Define the number of GPUs to be used in the training process
-    gpu_per_node = int(sys.argv[1])
-    gpu_num = 1
+    num_gpus = int(sys.argv[1])
 
     # Define and launch the multi-processing
 	import multiprocessing
     process = []
-    for gpu_num in range(0, gpu_per_node):
+    for gpu_num in range(0, num_gpus):
         process.append(multiprocessing.Process(target=train_mnist_cnn,
-                       args=(nccl_id, gpu_num, gpu_per_node)))
+                       args=(nccl_id, gpu_num, num_gpus)))
 
     for p in process:
         p.start()
 ```
 
+Here are some explanations concerning the variables created above:
+
+(i) `nccl_id`
+
+Note that we need to generate a NCCL ID here to be used for collective communication, and then pass it to all the processes. 
+The NCCL ID is like a ticket, where only the processes with this ID can join the AllReduce operation. 
+(Later if we use MPI, the passing of NCCL ID is not necessary, because the ID is broadcased by MPI in our code automatically)
+
+(ii) `num_gpus`
 
 Review comment:
   shall we simplify the terminology by using local_rank and global_rank?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services