[GitHub] solin319 opened a new issue #7554: There is a problem in distribute training with dtype=fp16

[gi...@git.apache.org]

solin319 opened a new issue #7554: There is a problem in distribute training with dtype=fp16
URL: https://github.com/apache/incubator-mxnet/issues/7554
 
 
   1. There is a problem in distribute training with dtype=fp16. The command is as below.
   
   python ../../tools/launch.py -n 2 --launcher ssh -H hosts `which python` train_imagenet.py \
   --data-train=/data/ILSVRC2012_img_train.rec \
   --data-val=/data/ILSVRC2012_img_val.rec \
   --network=resnet \
   --num-layers=50 \
   --kv-store=dist_sync \
   --gpus=0,1,2,3 \
   --disp-batch=10 \
   --top-k=5 \
   --batch-size=128 \
   --dtype='float16'
   
   Then the program break, and "Bad address (src/tcp.cpp:199)" was printed on the screen.
   
   2. I think the 'kvstore_dist.h' doesn't support fp16 yet. In the function 'Push_', the type of 'send_buf' is fp16. But the type of the data pointer passed to ps-lite is real_t(fp32).  There is the same problem was also in the function 'Pull'.
   Does these problem cause the break in distribute training with fp16?
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services