You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/05/19 03:47:08 UTC
[GitHub] [incubator-mxnet] XiaotaoChen commented on issue #18296: discussion about implementing SyncBN with nccl instead of shared cpu memory
XiaotaoChen commented on issue #18296:
URL: https://github.com/apache/incubator-mxnet/issues/18296#issuecomment-630561732
I sovle above problems by using cuda stream from mxnet by `mshadow::Stream<gpu>::GetStream(ctx.get_stream<gpu>())`, instead of create it. using `cuda stream` from mxnet guarantes all operations is done before doing allreduce. And remove the safeQueue, which is not safe in this scene. Currently, **SyncBN v3 is available.** and its' performance and result as belows:
### env
**hardware**
```shell
8gpu: 2080Ti
CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz 96 cores
```
**software**
```shell
simpledet: https://github.com/TuSimple/simpledet
configuration bases on config/resnet_v1b/faster_r50v1b_fpn_1x.py
adjustments:
1. normalizer: replaces "fixbn" with "syncbn" or "syncbn_v3"
2. fixed_param: replaces ["conv0", "stage1", "gamma", "beta"] with []
```
### performance and correctness
**profiling result**
| type | forward(ms/calls) | backward | calls |
| ----------------------------------------- | ------------------- | -------- | ----- |
| BatchNorm | 0.41 | 0.31 | 50400 |
| SyncBN | 0.87 | 4.01 | 50400 |
| syncBN with nccl | 1.28 | 2.50 | 50400 |
| syncBN with nccl merge_tensor (SyncBN v3) | 1.16 | 2.31 | 50400 |
It shows syncBN with nccl can improve 22% based on original SyncBN. And `syncBN with nccl+ merge tensor` can improve 28% based on original SyncBN. `merge tensor` means fusing `mean` and `var` into bigger tensor in forward stage(sumGrad, sumProd in backward stage), then call `allreduce` only once, instead of `call allreduce` per tensor.
**performance and result in real training scene**
| type | speed(samples/s) | IoU=0.5:0.95 | IoU=0.5 | IoU=0.75 |
| ----------------------------- | ---------------- | ------------ | ------- | -------- |
| syncBN with shared cpu memory | 24.40 | 0.372 | 0.594 | 0.4 |
| syncBN with nccl (SyncBN v3) | 32.32 | 0.370 | 0.591 | 0.397 |
### another question
above implemention of `SyncBN_v3` guarantes the synchronization between threads. However, it seems like mxnet has guaranted the synchronization between threads with its' stream. when i remove the sync code in `GlobalComm.reduce`, and call `ncclAllReduce` directly, the result is correct too. and its performance and result as belows:
| type | speed(samples/s) | IoU=0.5:0.95 | IoU=0.5 | IoU=0.75 |
| -------------------------------- | ---------------- | ------------ | ------- | -------- |
| syncBN with shared cpu memory | 24.40 | 0.372 | 0.594 | 0.4 |
| syncBN with nccl | 32.32 | 0.370 | 0.591 | 0.397 |
| syncBN with nccl no threads sync | 32.28 | 0.371 | 0.593 | 0.401 |
It shows that syncBN with nccl without threads sync is also correct. And its' performance is close to `syncBN with nccl with threads sync`. **So is synchronization between threads necessary for in `SyncBN v3` ?**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org