You are viewing a plain text version of this content. The canonical link for it is here.
Posted to devnull@infra.apache.org by "eric-haibin-lin (GitHub)" <gi...@apache.org> on 2018/08/26 16:34:46 UTC
[GitHub] [incubator-mxnet] eric-haibin-lin opened issue #12363:
distributed training notebook tests
How to leverage the existing tutorial test suite for tutorials for distributed training is not straightforward. Distributed training usually involves launcher scripts and multiple processes, as mentioned in https://github.com/apache/incubator-mxnet/pull/10955
@indhub @ThomasDelteil @sandeep-krishnamurthy
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] vishaalkapoor commented on issue #12363:
distributed training notebook tests
Posted by "vishaalkapoor (GitHub)" <gi...@apache.org>.
@sandeep-krishnamurthy It looks like the tutorial/notebook executor will need to be wrapped by a launcher script that executes notebooks in parallel on several nodes.
I modified the existing tutorial/notebook executor so that it could be used for more than tutorials on a single-host. There is no scaffolding for multi-host. https://github.com/apache/incubator-mxnet/blob/ae5d60fa830090f4882a433d9b88c53c26c42b4f/tests/utils/notebook_test/__init__.py#L39
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
Thank U, But if installed mxnet by pip, How to use python launch.py since I didn't download the mxnet source code?
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
I don't think so because I tried:
python /home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/mxnet/tools/launch.py -n 2 -s 2 -H hosts --sync-dst-dir /home/xiaomin.wu/cifar10_dist --launcher ssh "python /autofs/data56/public/xiaomin.wu/code/dist_mxnet_test/cifar10_dist.py"
the same errors show up
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
first of all, check out:https://github.com/apache/incubator-mxnet/tree/master/example/distributed_training
install anaconda on all your machines
run "pip install mxnet-cu80==1.2.1" on your all machines (or pip install mxnet-cu90 depends on your machine's env)
make all your machines sshable without input password between each other
machine A to machine B
cd ~/.ssh
ssh-keygen -t rsa
two files are generated: id_rsa is the secret key；and id_rsa.pub is the public key
in Machine B: vim ~/.ssh/authorized_keys, copy the contents in machine A:~/.ssh/id_rsa.pub here
do the same things to make machine B to machine A sshable without password
also， machine A to Machine A， machine B to machine B is needed
then run
python /home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/mxnet/tools/launch.py -n 2 -s 2 -H hosts --sync-dst-dir /home/xiaomin.wu/cifar10_dist --launcher ssh "/home/xiaomin.wu/anaconda2/bin/python cifar10_dist.py"
here we use /home/xiaomin.wu/anaconda2/bin/python instead of python, because if we just ues python here the machines may use /usr/bin/pythob, which will get you crazy.
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] ankkhedia commented on issue #12363:
distributed training notebook tests
Posted by "ankkhedia (GitHub)" <gi...@apache.org>.
@mxnet-label-bot [Distributed]
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
the machines are already sshable without password between each other
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
@eric-haibin-lin 我可以从a无密码ssh到b，也可以从b无密码ssh到a，但不能a无密码ssh到a，b也不能无密码ssh到b。而问题应该正是如此
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] YaoC commented on issue #12363:
distributed training notebook tests
Posted by "YaoC (GitHub)" <gi...@apache.org>.
```Python
# Create a distributed key-value store
store = kv.create('dist')
```
hi, is it `dist` the proper param of `kv.create`? As it is not in the parameters list.
https://mxnet.incubator.apache.org/versions/master/api/python/kvstore/kvstore.html#mxnet.kvstore.create
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
I download the mxnet source code, but didn't compile since i used the mxnet installed by pip. when i use the launch.py provided in the mxnet source code, some errors shows up:
xiaomin.wu@iva0605:/autofs/data56/public/xiaomin.wu/code/dist_mxnet_test$ python /autofs/data56/public/xiaomin.wu/software/incubator-mxnet-1.2.0/tools/launch.py -n 2 -s 2 -H hosts --sync-dst-dir /home/xiaomin.wu/cifar10_dist --launcher ssh "python /autofs/data56/public/xiaomin.wu/code/dist_mxnet_test/cifar10_dist.py"
Can't load dmlc_tracker package. Perhaps you need to run
git submodule update --init --recursive
Traceback (most recent call last):
File "/autofs/data56/public/xiaomin.wu/software/incubator-mxnet-1.2.0/tools/launch.py", line 128, in <module>
main()
File "/autofs/data56/public/xiaomin.wu/software/incubator-mxnet-1.2.0/tools/launch.py", line 96, in main
args = dmlc_opts(args)
File "/autofs/data56/public/xiaomin.wu/software/incubator-mxnet-1.2.0/tools/launch.py", line 48, in dmlc_opts
from dmlc_tracker import opts
ImportError: No module named dmlc_tracker
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
I update the version to 1.2.1 by pip, but new error shows up:(
xiaomin.wu@iva0605:/autofs/data56/public/xiaomin.wu/code/dist_mxnet_test$ python /home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/mxnet/tools/launch.py -n 2 -s 2 -H hosts --sync-dst-dir /home/xiaomin.wu/cifar10_dist --launcher ssh "python /autofs/data56/public/xiaomin.wu/code/dist_mxnet_test/cifar10_dist.py"
2018-08-27 15:54:12,883 INFO rsync /autofs/data56/public/xiaomin.wu/code/dist_mxnet_test/ -> 10.14.6.5:/home/xiaomin.wu/cifar10_dist
xiaomin.wu@10.14.6.5's password: /home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
2018-08-27 15:54:18,462 INFO rsync /autofs/data56/public/xiaomin.wu/code/dist_mxnet_test/ -> 10.14.6.8:/home/xiaomin.wu/cifar10_dist
xiaomin.wu@10.14.6.5's password: xiaomin.wu@10.14.6.5's password: Traceback (most recent call last):
File "/autofs/data56/public/xiaomin.wu/code/dist_mxnet_test/cifar10_dist.py", line 23, in <module>
import mxnet as mx
ImportError: No module named mxnet
Traceback (most recent call last):
File "/autofs/data56/public/xiaomin.wu/code/dist_mxnet_test/cifar10_dist.py", line 23, in <module>
import mxnet as mx
ImportError: No module named mxnet
Exception in thread Thread-5:
Traceback (most recent call last):
File "/home/xiaomin.wu/anaconda2/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/home/xiaomin.wu/anaconda2/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/dmlc_tracker/ssh.py", line 61, in run
subprocess.check_call(prog, shell = True)
File "/home/xiaomin.wu/anaconda2/lib/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'ssh -o StrictHostKeyChecking=no 10.14.6.8 -p 22 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64::/usr/local/cuda-8.0/lib64:~/TensorRT-4.0.0.3/lib; export DMLC_ROLE=worker; export DMLC_PS_ROOT_PORT=9093; export DMLC_PS_ROOT_URI=10.14.6.5; export DMLC_NUM_SERVER=2; export DMLC_NUM_WORKER=2; cd /home/xiaomin.wu/cifar10_dist; python /autofs/data56/public/xiaomin.wu/code/dist_mxnet_test/cifar10_dist.py'' returned non-zero exit status 1
Exception in thread Thread-3:
Traceback (most recent call last):
File "/home/xiaomin.wu/anaconda2/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/home/xiaomin.wu/anaconda2/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/dmlc_tracker/ssh.py", line 61, in run
subprocess.check_call(prog, shell = True)
File "/home/xiaomin.wu/anaconda2/lib/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'ssh -o StrictHostKeyChecking=no 10.14.6.8 -p 22 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64::/usr/local/cuda-8.0/lib64:~/TensorRT-4.0.0.3/lib; export DMLC_ROLE=server; export DMLC_PS_ROOT_PORT=9093; export DMLC_PS_ROOT_URI=10.14.6.5; export DMLC_NUM_SERVER=2; export DMLC_NUM_WORKER=2; cd /home/xiaomin.wu/cifar10_dist; python /autofs/data56/public/xiaomin.wu/code/dist_mxnet_test/cifar10_dist.py'' returned non-zero exit status 1
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on issue
#12363: distributed training notebook tests
Posted by "sandeep-krishnamurthy (GitHub)" <gi...@apache.org>.
@vishaalkapoor - Can we utilize your straightdope notebook testing setup here?
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #12363:
distributed training notebook tests
Posted by "szha (GitHub)" <gi...@apache.org>.
dist kvstore is already supported in pip.
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
Is it must be installed by source with USE_DIST_KVSTORE=1? Installed by pip didn't support distributed training?
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #12363:
distributed training notebook tests
Posted by "szha (GitHub)" <gi...@apache.org>.
could you upgrade your pip installation to 1.2.1post1?
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
hi, thankU guys, I solved the problemsm here is my solution:
https://shimo.im/docs/JwobIyIK8ucMgc3r/
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] vishaalkapoor commented on issue #12363:
distributed training notebook tests
Posted by "vishaalkapoor (GitHub)" <gi...@apache.org>.
@sandeep-krishnamurthy It looks like the tutorial/notebook executor will need to be wrapped by a launcher script that executes notebooks in parallel on several nodes.
I modified the existing tutorial/notebook executor so that it could be used for more than tutorials (arbitrary notebooks) on a single-host. There is no scaffolding for multi-host. https://github.com/apache/incubator-mxnet/blob/ae5d60fa830090f4882a433d9b88c53c26c42b4f/tests/utils/notebook_test/__init__.py#L39
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
the vesion of the mxnet i installed buy pip is 1.2.0
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] eric-haibin-lin commented on issue #12363:
distributed training notebook tests
Posted by "eric-haibin-lin (GitHub)" <gi...@apache.org>.
@meanmee looks like there's some problem connecting to your remote instance. Did you setup passwordless ssh? I suggest you move the question to discuss.mxnet.io which is monitored actively. Github issue is more for issue/bug report or task/feature requests.
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] eric-haibin-lin commented on issue #12363:
distributed training notebook tests
Posted by "eric-haibin-lin (GitHub)" <gi...@apache.org>.
pip package is already built with USE_DIST_KVSTORE=1. cc @szha in case I'm wrong
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] meanmee commented on issue #12363:
distributed training notebook tests
Posted by "meanmee (GitHub)" <gi...@apache.org>.
Permission denied, please try again.
xiaomin.wu@10.14.6.5's password:
Permission denied, please try again.
xiaomin.wu@10.14.6.5's password:
Permission denied (publickey,password).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.1]
Traceback (most recent call last):
File "/home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/mxnet/tools/launch.py", line 128, in <module>
main()
File "/home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/mxnet/tools/launch.py", line 113, in main
ssh.submit(args)
File "/home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/dmlc_tracker/ssh.py", line 86, in submit
hostIP=args.host_ip)
File "/home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/dmlc_tracker/tracker.py", line 428, in submit
fun_submit(nworker, nserver, envs)
File "/home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/dmlc_tracker/ssh.py", line 69, in ssh_submit
sync_dir(local_dir, h, working_dir)
File "/home/xiaomin.wu/anaconda2/lib/python2.7/site-packages/dmlc_tracker/ssh.py", line 21, in sync_dir
subprocess.check_call([prog], shell = True)
File "/home/xiaomin.wu/anaconda2/lib/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['rsync -az --rsh="ssh -o StrictHostKeyChecking=no -p 22" /home/xiaomin.wu/cifar10_dist/ 10.14.6.5:/home/xiaomin.wu/cifar10_dist']' returned non-zero exit status 255
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [incubator-mxnet] eric-haibin-lin commented on issue #12363:
distributed training notebook tests
Posted by "eric-haibin-lin (GitHub)" <gi...@apache.org>.
'dist' is equivalent to 'dist_sync'. Using the documented options is recommended.
[ Full content available at: https://github.com/apache/incubator-mxnet/issues/12363 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org