You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by gi...@git.apache.org on 2017/09/04 18:35:28 UTC
[GitHub] xcgoner opened a new issue #7726: Accuracy does not increase when using distributed kvstore
xcgoner opened a new issue #7726: Accuracy does not increase when using distributed kvstore
URL: https://github.com/apache/incubator-mxnet/issues/7726
Accuracy does not increase when using distributed kvstore. The accuracy fluctuate around 0.1
@sxjscience Can you help to inform the guys related to the distributed training?
## Environment info
Operating System:
Ubuntu 16.04.3 LTS
Compiler:
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Package used (Python/R/Scala/Julia):
Python
MXNet version:
Or if installed from source:
MXNet commit hash (`git rev-parse HEAD`):
d8e92df0fd80f074188b9c231e6bddcdf4cced46
If you are using python package, please provide
Python version and distribution:
Python 2.7.12
If you are using R package, please provide
R `sessionInfo()`:
## Error Message:
When using non-distributed version:
(mxnet-gpu-test) cx2@vision-gpu-1:/data/cx2/virtualenv/mxnet-gpu-test/src/incubator-mxnet/example/im
age-classification$ python train_mnist.py
INFO:root:start with arguments Namespace(add_stn=False, batch_size=64, disp_batches=100, dtype='floa
t32', gpus=None, kv_store='device', load_epoch=None, lr=0.05, lr_factor=0.1, lr_step_epochs='10', mo
del_prefix=None, mom=0.9, monitor=0, network='mlp', num_classes=10, num_epochs=20, num_examples=6000
0, num_layers=None, optimizer='sgd', test_io=0, top_k=0, wd=0.0001)
INFO:root:Epoch[0] Batch [100] Speed: 1682.44 samples/sec accuracy=0.768719
INFO:root:Epoch[0] Batch [200] Speed: 1621.89 samples/sec accuracy=0.905000
INFO:root:Epoch[0] Batch [300] Speed: 1730.96 samples/sec accuracy=0.929375
INFO:root:Epoch[0] Batch [400] Speed: 1817.02 samples/sec accuracy=0.937344
INFO:root:Epoch[0] Batch [500] Speed: 2072.11 samples/sec accuracy=0.940156
INFO:root:Epoch[0] Batch [600] Speed: 2183.72 samples/sec accuracy=0.948750
INFO:root:Epoch[0] Batch [700] Speed: 1955.86 samples/sec accuracy=0.954688
INFO:root:Epoch[0] Batch [800] Speed: 1764.03 samples/sec accuracy=0.951875
INFO:root:Epoch[0] Batch [900] Speed: 2086.64 samples/sec accuracy=0.956562
INFO:root:Epoch[0] Train-accuracy=0.959882
INFO:root:Epoch[0] Time cost=34.679
INFO:root:Epoch[0] Validation-accuracy=0.950936
INFO:root:Epoch[1] Batch [100] Speed: 2131.35 samples/sec accuracy=0.955755
INFO:root:Epoch[1] Batch [200] Speed: 2316.70 samples/sec accuracy=0.959219
INFO:root:Epoch[1] Batch [300] Speed: 2100.69 samples/sec accuracy=0.965313
When using distributed version with single process:
(mxnet-gpu-test) cx2@vision-gpu-1:/data/cx2/virtualenv/mxnet-gpu-test/src/incubator-mxnet/example/im
age-classification$ python ../../tools/launch.py -n 1 --launcher ssh -H hosts python train_mnist.py
--kv-store dist_sync
INFO:root:start with arguments Namespace(add_stn=False, batch_size=64, disp_batches=100, dtype='floa
t32', gpus=None, kv_store='dist_sync', load_epoch=None, lr=0.05, lr_factor=0.1, lr_step_epochs='10',
model_prefix=None, mom=0.9, monitor=0, network='mlp', num_classes=10, num_epochs=20, num_examples=6
0000, num_layers=None, optimizer='sgd', test_io=0, top_k=0, wd=0.0001)
INFO:root:Epoch[0] Batch [100] Speed: 1570.26 samples/sec accuracy=0.097308
INFO:root:Epoch[0] Batch [200] Speed: 1684.40 samples/sec accuracy=0.101719
INFO:root:Epoch[0] Batch [300] Speed: 1967.76 samples/sec accuracy=0.094531
INFO:root:Epoch[0] Batch [400] Speed: 1448.44 samples/sec accuracy=0.100469
INFO:root:Epoch[0] Batch [500] Speed: 1653.76 samples/sec accuracy=0.093125
INFO:root:Epoch[0] Batch [600] Speed: 1659.77 samples/sec accuracy=0.100937
INFO:root:Epoch[0] Batch [700] Speed: 1776.14 samples/sec accuracy=0.100625
INFO:root:Epoch[0] Batch [800] Speed: 1707.80 samples/sec accuracy=0.098437
INFO:root:Epoch[0] Batch [900] Speed: 1968.17 samples/sec accuracy=0.100781
INFO:root:Epoch[0] Train-accuracy=0.098395
INFO:root:Epoch[0] Time cost=35.276
INFO:root:Epoch[0] Validation-accuracy=0.098029
INFO:root:Epoch[1] Batch [100] Speed: 1615.71 samples/sec accuracy=0.097772
INFO:root:Epoch[1] Batch [200] Speed: 1660.07 samples/sec accuracy=0.101719
INFO:root:Epoch[1] Batch [300] Speed: 1997.55 samples/sec accuracy=0.094531
When using distributed version with 2 processes:
(mxnet-gpu-test) cx2@vision-gpu-1:/data/cx2/virtualenv/mxnet-gpu-test/src/incubator-mxnet/example/im
age-classification$ python ../../tools/launch.py -n 2 --launcher ssh -H hosts python train_mnist.py
--kv-store dist_sync
INFO:root:start with arguments Namespace(add_stn=False, batch_size=64, disp_batches=100, dtype='floa
t32', gpus=None, kv_store='dist_sync', load_epoch=None, lr=0.05, lr_factor=0.1, lr_step_epochs='10',
model_prefix=None, mom=0.9, monitor=0, network='mlp', num_classes=10, num_epochs=20, num_examples=6
0000, num_layers=None, optimizer='sgd', test_io=0, top_k=0, wd=0.0001)
INFO:root:start with arguments Namespace(add_stn=False, batch_size=64, disp_batches=100, dtype='floa
t32', gpus=None, kv_store='dist_sync', load_epoch=None, lr=0.05, lr_factor=0.1, lr_step_epochs='10',
model_prefix=None, mom=0.9, monitor=0, network='mlp', num_classes=10, num_epochs=20, num_examples=6
0000, num_layers=None, optimizer='sgd', test_io=0, top_k=0, wd=0.0001)
INFO:root:Epoch[0] Batch [100] Speed: 865.23 samples/sec accuracy=0.099783
INFO:root:Epoch[0] Batch [100] Speed: 755.68 samples/sec accuracy=0.099319
INFO:root:Epoch[0] Batch [200] Speed: 886.68 samples/sec accuracy=0.101250
INFO:root:Epoch[0] Batch [200] Speed: 775.73 samples/sec accuracy=0.100937
INFO:root:Epoch[0] Batch [300] Speed: 995.09 samples/sec accuracy=0.097812
INFO:root:Epoch[0] Batch [300] Speed: 955.45 samples/sec accuracy=0.093281
INFO:root:Epoch[0] Batch [400] Speed: 942.12 samples/sec accuracy=0.097187
INFO:root:Epoch[0] Batch [400] Speed: 927.82 samples/sec accuracy=0.099062
INFO:root:Epoch[0] Batch [500] Speed: 1142.68 samples/sec accuracy=0.095000
INFO:root:Epoch[0] Batch [500] Speed: 1207.60 samples/sec accuracy=0.098750
INFO:root:Epoch[0] Batch [600] Speed: 1057.95 samples/sec accuracy=0.095156
INFO:root:Epoch[0] Batch [600] Speed: 1058.24 samples/sec accuracy=0.098281
INFO:root:Epoch[0] Batch [700] Speed: 1047.78 samples/sec accuracy=0.102656
INFO:root:Epoch[0] Batch [700] Speed: 974.27 samples/sec accuracy=0.099219
## Minimum reproducible example
Using the original example train_mnist
## Steps to reproduce
or if you are running standard examples, please provide the commands you have run that lead to the error.
1. Compile:
make -j5 USE_OPENCV=0 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_DIST_KVSTORE=1
2. hosts: 127.0.0.1
3. non-distributed: python train_mnist.py
4. distributed with single process: python ../../tools/launch.py -n 1 --launcher ssh -H hosts python train_mnist.py --kv-store dist_sync
5. distributed with 2 processes: python ../../tools/launch.py -n 2 --launcher ssh -H hosts python train_mnist.py --kv-store dist_sync
## What have you tried to solve it?
Not yet.
However, the version 0.11.0.rc3 does not have this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services