[PR] [3rdparty] AUTO mode for custom all-reduce strategy [tvm]

["MasterJH5574 (via GitHub)" <gi...@apache.org>]

MasterJH5574 opened a new pull request, #16797:
URL: https://github.com/apache/tvm/pull/16797

   This PR adds the automatic mode selection for customized all-reduce kernels, referring TensorRT-LLM.
   
   Meanwhile, this PR fixes a bug that may cause customized all-reduce kernel to hang forever. Prior to this PR, each worker resets its barrier values to 0 *after using all-gather to exchange their barrier handles*. Afterwards, the customized all-reduce kernels update the barriers of all workers. So it is possible that, worker 0 updates worker 1's barrier *before* worker 1 resets its barrier to 0. This lead to the all-reduce kernel hanging forever.
   
   This PR changes the behavior to resetting barriers before all-gather, and forcing a device synchronization after reset.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [3rdparty] AUTO mode for custom all-reduce strategy [tvm]

["yongwww (via GitHub)" <gi...@apache.org>]

yongwww merged PR #16797:
URL: https://github.com/apache/tvm/pull/16797


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org