You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/05/16 11:00:10 UTC
[GitHub] [incubator-mxnet] sxjscience commented on issue #14935: [OP]
Accelerate GPU version of LayerNorm(axis=-1)
sxjscience commented on issue #14935: [OP] Accelerate GPU version of LayerNorm(axis=-1)
URL: https://github.com/apache/incubator-mxnet/pull/14935#issuecomment-493021248
I tested the speed of LayerNorm(axis=-1) with different batch, channel, dtype combinations. The results are listed as follows. We use both `nvprof` to profile the speed of the kernels and `time.time()` to check the time spent in the python side.
For the implementation in apex (https://github.com/NVIDIA/apex) + FP32
PyTorch + Apex
Forward (nvprof timer)
| | B=128 | B=2560 | B=4096 | B=8192 | B=16384 |
| --- | --- | --- | --- | --- | --- |
|**C=32**| 8 | 18.6 | 31.3 | 55.5 | 105 |
|**C=64**| 8.4 | 18.8 | 31.4 | 55.7 | 107.7 |
|**C=128**| 6.3 | 14.4 | 22.4 | 41.3 | 76.7 |
|**C=256**| 6.2 | 16.9 | 26.8 | 47.9 | 91.1 |
|**C=512**| 6.8 | 26 | 37.8 | 69.4 | 132.8 |
|**C=768**| 8.1 | 30.1 | **44.5** | **81.5** | 156.1 |
|**C=1024**| 8.7 | 37.9 | **57.2** | **106.2** | 204.4 |
Backward (nvprof timer)
| | B=128 | B=2560 | B=4096 | B=8192 | B=16384 |
| --- | --- | --- | --- | --- | --- |
|**C=32**| 2.4 | 7.4 | 10.8 | 19.4 | 37.4 |
|**C=64**| 2.7 | 7.7 | 11.1 | 20.6 | 40.5 |
|**C=128**| 2.7 | 8.2 | 12.3 | 23.2 | 44.8 |
|**C=256**| 3.4 | 11.1 | 16.4 | 31.7 | 63.2 |
|**C=512**| 3.8 | 18.7 | 31.3 | 62.9 | 127.5 |
|**C=768**| 4.4 | 30 | **50** | **100.9** | 200.3 |
|**C=1024**| 4.9 | 45.3 | **71.6** | **144.2** | 284.4 |
Backward Data (nvprof timer)
| | B=128 | B=2560 | B=4096 | B=8192 | B=16384 |
| --- | --- | --- | --- | --- | --- |
|**C=32**| 2.4 | 7.4 | 10.8 | 19.4 | 37.4 |
|**C=64**| 2.7 | 7.7 | 11.1 | 20.6 | 40.5 |
|**C=128**| 2.7 | 8.2 | 12.3 | 23.2 | 44.8 |
|**C=256**| 3.4 | 11.1 | 16.4 | 31.7 | 63.2 |
|**C=512**| 3.8 | 18.7 | 31.3 | 62.9 | 127.5 |
|**C=768**| 4.4 | 30 | **50** | **100.9** | 200.3 |
|**C=1024**| 4.9 | 45.3 | **71.6** | **144.2** | 284.4 |
Backward Gamma & Beta (nvprof timer)
| | B=128 | B=2560 | B=4096 | B=8192 | B=16384 |
| --- | --- | --- | --- | --- | --- |
|**C=32**| 5.2 | 10.6 | 13.9 | 22.8 | 52.9 |
|**C=64**| 5.2 | 10.7 | 14 | 29.5 | 58 |
|**C=128**| 5.3 | 10.7 | 18.1 | 35 | 60.1 |
|**C=256**| 5.3 | 16.5 | 24.2 | 39.9 | 68.1 |
|**C=512**| 5.7 | 24.9 | 32.8 | 52.6 | 90.7 |
|**C=768**| 6 | 31.5 | **42.5** | **71.5** | 128.1 |
|**C=1024**| 6.4 | 36.8 | **50** | **89.3** | 164.7 |
Forward (python timer)
| | B=128 | B=2560 | B=4096 | B=8192 | B=16384 |
| --- | --- | --- | --- | --- | --- |
|**C=32**| 109.7 | 106.3 | 105.9 | 120.2 | 157.8 |
|**C=64**| 104.8 | 106.5 | 107.4 | 122.1 | 169.4 |
|**C=128**| 103.6 | 107 | 110.2 | 119.8 | 170 |
|**C=256**| 103.2 | 109.8 | 115.9 | 152 | 210.4 |
|**C=512**| 105.1 | 135.8 | 149.7 | 195 | 243.8 |
|**C=768**| 105.5 | 144.4 | **170.8** | **206.2** | 258.2 |
|**C=1024**| 105.4 | 163.2 | **189.4** | **228.7** | 291.9 |
Backward (python timer)
| | B=128 | B=2560 | B=4096 | B=8192 | B=16384 |
| --- | --- | --- | --- | --- | --- |
|**C=32**| 237.1 | 224.6 | 241.2 | 239.1 | 246.4 |
|**C=64**| 233.5 | 216.9 | 223.2 | 226.9 | 235.4 |
|**C=128**| 207.7 | 232.1 | 240.2 | 245 | 273.1 |
|**C=256**| 217.6 | 240.6 | 251.4 | 277 | 309 |
|**C=512**| 238.3 | 267 | 278.8 | 300.2 | 315.5 |
|**C=768**| 216 | 287.1 | **285.8** | **304** | 387.7 |
|**C=1024**| 217.8 | 279.7 | **315.8** | **330.4** | 472.5 |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services