[GitHub] sssun0228 opened a new issue #10057: Multiple workers on single CPU training

[GitBox <gi...@apache.org>]

sssun0228 opened a new issue #10057: Multiple workers on single CPU training
URL: https://github.com/apache/incubator-mxnet/issues/10057
 
 
   Hi,
   
   We are trying to spawn multiple workers on single CPU because of under-utilization of the CPU cores during the training. So we used local kvstore first as follows. And the hosts file contains 2 lines of the duplicated IP address.
   
   _~/mxnet/tools/launch.py -n 2 --launcher ssh -H ./hosts numactl -l python train_imagenet.py --kv-store local_
   
   After 1 epoch we compared the models from each workers and found that workers don't synchronize parameters. Next, we tried dist_sync with the same command and it gave following error.
   
     _File "/home/.local/lib/python2.7/site-packages/mxnet-1.2.0-py2.7.egg/mxnet/kvstore.py", line 655, in create
       ctypes.byref(handle)))
     File "/home/.local/lib/python2.7/site-packages/mxnet-1.2.0-py2.7.egg/mxnet/base.py", line 149, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [18:08:10] src/postoffice.cc:16: Check  notnull: Environment::Get()->find("DMLC_NUM_WORKER")_ 
   
    If we change the launcher to mpi, there is the following error.
   _Traceback (most recent call last):
     File "~/mxnet/tools/launch.py", line 128, in <module>
       main()
     File "~/mxnet/tools/launch.py", line 116, in main
       mpi.submit(args)
     File "~/mxnet/tools/../dmlc-core/tracker/dmlc_tracker/mpi.py", line 82, in submit
       pscmd=(' '.join(args.command)))
     File "~/mxnet/tools/../dmlc-core/tracker/dmlc_tracker/tracker.py", line 428, in submit
       fun_submit(nworker, nserver, envs)
     File "~/mxnet/tools/../dmlc-core/tracker/dmlc_tracker/mpi.py", line 61, in mpi_submit
       prog = 'mpirun -n %d %s %s' % (nworker, get_mpi_env(pass_envs), cmd)
     File "~/mxnet/tools/../dmlc-core/tracker/dmlc_tracker/mpi.py", line 35, in get_mpi_env
       raise RuntimeError('Unknown MPI Version')_
   
   Do you support for multi workers/node training? Any comments are appreciated. Thanks,
   
   
   Compiler (gcc/clang/mingw/visual studio):
   gcc
   
   MXNet commit hash:
   02dd89a68f659c2a9b0bff62c54c50dff1151f6b
   
   Build config:
   USE_MKLDNN = 1
   USE_DIST_KVSTORE = 1
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services