You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/11/18 01:29:25 UTC
[GitHub] [incubator-mxnet] jonatan1626 opened a new issue #16845: MXNet
1.6.0 performance regression
jonatan1626 opened a new issue #16845: MXNet 1.6.0 performance regression
URL: https://github.com/apache/incubator-mxnet/issues/16845
## Description
I wanted to report an issue with the performance of mxnet 1.6.x when comparing it to the previous version, mxnet 1.5.x
p2.16xlarge | | | | | | | |
-- | -- | -- | -- | -- | -- | -- | -- | --
MXnet Version | Model | Num GPU | avg_speed | min_speed | max_speed | avg_gpu_memory | min_gpu_memory | max_gpu_memory
mxnet1.5 | resnet101_v1 | num_gpu_1 | 279.82 | 271.97 | 285.07 | 1269.93 | 220.00 | 10037.00
mxnet1.6_x | resnet101_v1 | num_gpu_1 | 316.53 | 312.23 | 329.48 | 1261.46 | 1106.00 | 1465.00
**Percentage difference 1.6.x/1.5** | | | 1.131176404 | 1.148008905 | 1.155794134 | 0.993325863 | 5.027272727 | 0.145959948
mxnet1.5 | resnet152_v1 | num_gpu_1 | 386.45 | 365.09 | 392.36 | 1351.67 | 1341.00 | 1356.00
mxnet1.6_x | resnet152_v1 | num_gpu_1 | 433.14 | 418.71 | 462.85 | 1412.06 | 1392.00 | 1421.00
**Percentage difference 1.6.x/1.5** | | | 1.120815624 | 1.146855987 | 1.179660748 | 1.044674962 | 1.03803132 | 1.047935103
p3.16xlarge | | | | | | | |
-- | -- | -- | -- | -- | -- | -- | -- | --
MXnet Version | Model | Num GPU | avg_speed | min_speed | max_speed | avg_gpu_memory | min_gpu_memory | max_gpu_memory
mxnet1.5 | resnet101_v1 | num_gpu_1 | 194.04 | 164.38 | 201.67 | 1737.57 | 1734.00 | 1740.00
mxnet1.6_x | resnet101_v1 | num_gpu_1 | 200.38 | 189.29 | 225.85 | 1988.92 | 1824.00 | 2136.00
**Percentage difference 1.6.x/1.5** | | | 1.032671936 | 1.151536731 | 1.119874587 | 1.144658131 | 1.051903114 | 1.227586207
mxnet1.5 | resnet152_v1 | num_gpu_1 | 275.77 | 221.22 | 282.92 | 2016.00 | 2016.00 | 2016.00
mxnet1.6_x | resnet152_v1 | num_gpu_1 | 337.50 | 317.80 | 348.30 | 2134.69 | 2130.00 | 2136.00
**Percentage difference 1.6.x/1.5** | | | 1.223845898 | 1.436586371 | 1.231101633 | 1.058871723 | 1.056547619 | 1.05952381
## To Reproduce
1. Create a new conda enviornment for each version.
To install 1.5: `pip install mxnet-cu101mkl`
To install 1.6.x:
`checkout https://github.com/apeforest/mxnet-build-script`
`Change the version to v1.6.x in the Dockerfile`
`Follow instructions and launch and build mxnet-cu100`
`Copy pip wheel out of the container and pip install`
2. Here are the links for the scripts:
[Gluon Model Zoo Site for Cifar 10](https://gluon-cv.mxnet.io/build/examples_classification/dive_deep_cifar10.html)
[Link for the Cifar 10 Training Script](https://gluon-cv.mxnet.io/_downloads/b95defa36d5f5e2c64a2569f387ca7b2/train_cifar10.py)
3. Launching the scripts with the respective conda env with mxnet version installed:
Launch nvidia-smi to get GPU Memory Usage:
`nvidia-smi --query-gpu=index,memory.used --format=csv -l 30 -f <file_location>`
Launch model training:
`python train_cifar10.py --num-gpus 1 --model resnet152_v1 --num-epochs 40
`
`python train_cifar10.py --num-gpus 1 --model resnet101_v1 --num-epochs 40
`
The script will print out time it took per epoch.
I used this regex to match and grab the time after 3 epochs, treating those as the warmup.
`r'.*\[Epoch ([0-9]*).*\].* time: ([0-9]*\.?[0-9]*)'`
I then took the average, min, max values.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services