You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by gi...@git.apache.org on 2017/08/14 14:51:44 UTC
[GitHub] leoxiaobin opened a new issue #7455: Distributed training is slow
leoxiaobin opened a new issue #7455: Distributed training is slow
URL: https://github.com/apache/incubator-mxnet/issues/7455
## Environment info
Operating System: Ubuntu 16.4
Compiler: gcc 5.4
Package used (Python/R/Scala/Julia): Python
MXNet version: Last code
Or if installed from source: installed from source
MXNet commit hash (`git rev-parse HEAD`): 1a3faa
If you are using python package, please provide
Python version and distribution: Python 2.7.13 :: Anaconda custom (64-bit)
I tried to train image classification model using two servers with infiniband cards. But the speed is a little slow, just like using one server. I used the code of example/image-classifaction.
when training on one server, the training command is
```
python train_imagenet.py --benchmark 1 --gpus 0,1,2,3,4,5,6,7 --kv-store device --network inception-v3 --batch-size 256 --image-shape 3,299,299
```
the speed is
```
INFO:root:start with arguments Namespace(batch_size=256, benchmark=1, data_nthreads=4, data_train=None, data_val=None, disp_batches=20, dtype='float32', gpus='0,1,2,3,4,5,6,7', image_shape='3,299,299', kv_store='device', load_epoch=None, lr=0.1, lr_factor=0.1, lr_step_epochs='30,60', max_random_aspect_ratio=0.25, max_random_h=36, max_random_l=50, max_random_rotate_angle=10, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0.1, min_random_scale=1, model_prefix=None, mom=0.9, monitor=0, network='inception-v3', num_classes=1000, num_epochs=80, num_examples=1281167, num_layers=50, optimizer='sgd', pad_size=0, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)
[22:35:19] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[22:35:40] src/kvstore/././comm.h:327: only 24 out of 56 GPU pairs are enabled direct access. It may affect the performance. You can set MXNET_ENABLE_GPU_P2P=0 to turn it off
[22:35:40] src/kvstore/././comm.h:336: .vvv....
[22:35:40] src/kvstore/././comm.h:336: v.vv....
[22:35:40] src/kvstore/././comm.h:336: vv.v....
[22:35:40] src/kvstore/././comm.h:336: vvv.....
[22:35:40] src/kvstore/././comm.h:336: .....vvv
[22:35:40] src/kvstore/././comm.h:336: ....v.vv
[22:35:40] src/kvstore/././comm.h:336: ....vv.v
[22:35:40] src/kvstore/././comm.h:336: ....vvv.
INFO:root:Epoch[0] Batch [20] Speed: 1065.93 samples/sec accuracy=0.165365
INFO:root:Epoch[0] Batch [40] Speed: 1033.22 samples/sec accuracy=0.989648
INFO:root:Epoch[0] Batch [60] Speed: 1029.90 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [80] Speed: 1029.80 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [100] Speed: 1028.05 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [120] Speed: 1019.75 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [140] Speed: 1025.79 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [160] Speed: 1027.82 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [180] Speed: 1021.11 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [200] Speed: 1025.14 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [220] Speed: 1017.72 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [240] Speed: 1021.09 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [260] Speed: 1024.25 samples/sec accuracy=1.000000
```
When training with 2 servers, the command is
```
python ../../tools/launch.py -n 2 --launcher ssh -H hosts python train_imagenet.py --benchmark 1 --gpus 0,1,2,3,4,5,6,7 --kv-store dist_sync --network inception-v3 --num-layers 50 --batch-size 256 --sync-dst-dir /tmp/mxnet --image-shape 3,299,299
```
And the speed is
```
INFO:root:Epoch[0] Batch [20] Speed: 609.31 samples/sec accuracy=0.056920
INFO:root:Epoch[0] Batch [20] Speed: 610.12 samples/sec accuracy=0.050967
INFO:root:Epoch[0] Batch [40] Speed: 608.68 samples/sec accuracy=0.854883
INFO:root:Epoch[0] Batch [40] Speed: 608.19 samples/sec accuracy=0.868164
INFO:root:Epoch[0] Batch [60] Speed: 602.48 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [60] Speed: 603.86 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [80] Speed: 603.11 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [80] Speed: 603.87 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [100] Speed: 607.30 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [100] Speed: 606.54 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [120] Speed: 604.53 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [120] Speed: 602.63 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [140] Speed: 601.27 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [140] Speed: 603.67 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [160] Speed: 603.64 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [160] Speed: 602.81 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [180] Speed: 606.20 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [180] Speed: 606.28 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [200] Speed: 604.40 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [200] Speed: 604.28 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [220] Speed: 605.54 samples/sec accuracy=1.000000
INFO:root:Epoch[0] Batch [220] Speed: 605.21 samples/sec accuracy=1.000000
```
It seems the using distributed training with 2 servers, the speed is only a little better than standalone training(600x2 samples/sec VS 1000 samples/sec).
I tried to test the IB bandwidth using iperf, it can get 24.0 Gbits/sec using 1 thread, so I think the IB's bandwidth is not the bottleneck.
Does anyone can give any suggestion about distributed training using MxNet?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services