You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "LeshengJin (via GitHub)" <gi...@apache.org> on 2023/08/28 04:55:25 UTC
[GitHub] [tvm] LeshengJin opened a new pull request, #15633: [Disco][Op] broadcast_from_zero
LeshengJin opened a new pull request, #15633:
URL: https://github.com/apache/tvm/pull/15633
This pr introduces op broadcast_from_zero, which broadcasts input tensor from worker-0 to all other workers.
cc: @junrushao @jinhongyii @tqchen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tqchen commented on a diff in pull request #15633: [Disco][Op] broadcast_from_zero
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on code in PR #15633:
URL: https://github.com/apache/tvm/pull/15633#discussion_r1307738990
##########
src/relax/op/ccl/ccl.h:
##########
@@ -35,6 +35,9 @@ namespace relax {
/*! \brief AllReduce. */
Expr allreduce(Expr data, String op_type);
+/*! \brief Broadcast data from worker-0 to all other workers. */
+Expr broadcast_from_zero(Expr data);
Review Comment:
agree about broadcast_from_worker0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao merged pull request #15633: [Disco][Op] broadcast_from_worker0
Posted by "junrushao (via GitHub)" <gi...@apache.org>.
junrushao merged PR #15633:
URL: https://github.com/apache/tvm/pull/15633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #15633: [Disco][Op] broadcast_from_zero
Posted by "junrushao (via GitHub)" <gi...@apache.org>.
junrushao commented on code in PR #15633:
URL: https://github.com/apache/tvm/pull/15633#discussion_r1307739714
##########
src/relax/op/ccl/ccl.h:
##########
@@ -35,6 +35,9 @@ namespace relax {
/*! \brief AllReduce. */
Expr allreduce(Expr data, String op_type);
+/*! \brief Broadcast data from worker-0 to all other workers. */
+Expr broadcast_from_zero(Expr data);
Review Comment:
cool. @LeshengJin please update the PR to reflect the name consistency and let's get it merged today!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #15633: [Disco][Op] broadcast_from_zero
Posted by "junrushao (via GitHub)" <gi...@apache.org>.
junrushao commented on code in PR #15633:
URL: https://github.com/apache/tvm/pull/15633#discussion_r1307575087
##########
src/relax/op/ccl/ccl.h:
##########
@@ -35,6 +35,9 @@ namespace relax {
/*! \brief AllReduce. */
Expr allreduce(Expr data, String op_type);
+/*! \brief Broadcast data from worker-0 to all other workers. */
+Expr broadcast_from_zero(Expr data);
Review Comment:
It’s fine to have numbers
##########
src/relax/op/ccl/ccl.h:
##########
@@ -35,6 +35,9 @@ namespace relax {
/*! \brief AllReduce. */
Expr allreduce(Expr data, String op_type);
+/*! \brief Broadcast data from worker-0 to all other workers. */
+Expr broadcast_from_zero(Expr data);
Review Comment:
It’s fine to have numbers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #15633: [Disco][Op] broadcast_from_zero
Posted by "junrushao (via GitHub)" <gi...@apache.org>.
junrushao commented on code in PR #15633:
URL: https://github.com/apache/tvm/pull/15633#discussion_r1307124823
##########
src/relax/op/ccl/ccl.h:
##########
@@ -35,6 +35,9 @@ namespace relax {
/*! \brief AllReduce. */
Expr allreduce(Expr data, String op_type);
+/*! \brief Broadcast data from worker-0 to all other workers. */
+Expr broadcast_from_zero(Expr data);
Review Comment:
Let’s think of a consistent naming for this primitive across Relax and Disco/NCCL. broadcast_from_worker0 or broadcast_from_zero?
##########
src/relax/op/ccl/ccl.h:
##########
@@ -35,6 +35,9 @@ namespace relax {
/*! \brief AllReduce. */
Expr allreduce(Expr data, String op_type);
+/*! \brief Broadcast data from worker-0 to all other workers. */
+Expr broadcast_from_zero(Expr data);
Review Comment:
Let’s think of a consistent naming for this primitive across Relax and Disco/NCCL. broadcast_from_worker0 or broadcast_from_zero?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] junrushao commented on a diff in pull request #15633: [Disco][Op] broadcast_from_worker0
Posted by "junrushao (via GitHub)" <gi...@apache.org>.
junrushao commented on code in PR #15633:
URL: https://github.com/apache/tvm/pull/15633#discussion_r1307971528
##########
src/runtime/disco/nccl/nccl.cc:
##########
@@ -109,13 +109,14 @@ NDArray AllReduce(NDArray send, ReduceKind reduce_kind) {
return recv;
}
-void BroadcastFromZero(NDArray buffer) {
+NDArray BroadcastFromZero(NDArray buffer) {
Review Comment:
I'll later change its name to BroadcastFromWorker0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] LeshengJin commented on a diff in pull request #15633: [Disco][Op] broadcast_from_worker0
Posted by "LeshengJin (via GitHub)" <gi...@apache.org>.
LeshengJin commented on code in PR #15633:
URL: https://github.com/apache/tvm/pull/15633#discussion_r1307796184
##########
src/relax/op/ccl/ccl.h:
##########
@@ -35,6 +35,9 @@ namespace relax {
/*! \brief AllReduce. */
Expr allreduce(Expr data, String op_type);
+/*! \brief Broadcast data from worker-0 to all other workers. */
+Expr broadcast_from_zero(Expr data);
Review Comment:
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] LeshengJin commented on a diff in pull request #15633: [Disco][Op] broadcast_from_zero
Posted by "LeshengJin (via GitHub)" <gi...@apache.org>.
LeshengJin commented on code in PR #15633:
URL: https://github.com/apache/tvm/pull/15633#discussion_r1307177778
##########
src/relax/op/ccl/ccl.h:
##########
@@ -35,6 +35,9 @@ namespace relax {
/*! \brief AllReduce. */
Expr allreduce(Expr data, String op_type);
+/*! \brief Broadcast data from worker-0 to all other workers. */
+Expr broadcast_from_zero(Expr data);
Review Comment:
I prefer broadcast_from_worker0, which is more straightforward. But I'm not sure if it's ok to have numbers in function names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org