[GitHub] [incubator-mxnet] renganxu edited a comment on issue #14047: mxnet.base.MXNetError: Cannot find argument 'cudnn_algo_verbose'

[GitBox <gi...@apache.org>]

renganxu edited a comment on issue #14047: mxnet.base.MXNetError: Cannot find argument 'cudnn_algo_verbose'
URL: https://github.com/apache/incubator-mxnet/issues/14047#issuecomment-477673243
 
 
   Hi @ptrendx, since my system could not install Docker, I converted the mxnet Docker container to Singularity container and then used Singularity container instead. My detailed command is:
   ```
   mpirun --allow-run-as-root --tag-output -np 8 --map-by socket -report-bindings -mca pml ob1 -mca btl_openib_receive_queues X,4096,1024:X,12288,512:X,65536,512 \
   singularity exec -B /cm/local/apps/cuda-driver/libs/410.66:/mnt/driver -B /mnt/isilon:/mnt/isilon sandbox-mxnet-ngc18.11 bash container_cmd.sh
   ```
   The commands in the file container_cmd.sh is as follows:
   ```
   export MXNET_UPDATE_ON_KVSTORE=0
   export MXNET_CUDNN_AUTOTUNE_DEFAULT=0
   export MXNET_EXEC_ENABLE_ADDTO=1
   export MXNET_USE_TENSORRT=0
   export MXNET_GPU_WORKER_NTHREADS=1
   export MXNET_GPU_COPY_NTHREADS=1
   export MXNET_CUDNN_AUTOTUNE_DEFAULT=0
   export MXNET_OPTIMIZER_AGGREGATION_SIZE=54
   export NCCL_SOCKET_IFNAME=ib0
   export NCCL_BUFFSIZE=2097152
   export NCCL_NET_GDR_READ=1
   export HOROVOD_CYCLE_TIME=0.2
   export HOROVOD_TWO_STAGE_LOOP=1
   export HOROVOD_ALLREDUCE_MODE=1
   export HOROVOD_FIXED_PAYLOAD=161
   
   python train_imagenet.py \
   --gpus 0,1,2,3 \
   --batch-size 208 \
   --kv-store horovod \
   --lr 0.6 \
   --lr-step-epochs 30,60,80,100 \
   --warmup-epochs 5 \
   --eval-period 4 \
   --eval-offset 2      \
   --optimizer sgd \
   --network resnet-v1b-fl \
   --num-layers 50 \
   --num-epochs 121 \
   --accuracy-threshold 0.749 \
   --dtype float16 \
   --use-dali \
   --disp-batches 20 \
   --image-shape 4,224,224 \
   --fuse-bn-relu 1 \
   --fuse-bn-add-relu 1 \
   --min-random-area 0.05 \
   --max-random-area 1.0 \
   --conv-algo 1 \
   --force-tensor-core 1 \
   --input-layout NHWC \
   --conv-layout NHWC \
   --batchnorm-layout NHWC \
   --pooling-layout NHWC \
   --batchnorm-mom 0.9 \
   --batchnorm-eps 1e-5 \
   --data-train /mnt/isilon/DeepLearning/database/mlperf/ilsvrc12_recordio/train.rec \
   --data-train-idx /mnt/isilon/DeepLearning/database/mlperf/ilsvrc12_recordio/train.idx \
   --data-val /mnt/isilon/DeepLearning/database/mlperf/ilsvrc12_recordio/val.rec \
   --data-val-idx /mnt/isilon/DeepLearning/database/mlperf/ilsvrc12_recordio/val.idx \
   --dali-prefetch-queue 2 \
   --dali-nvjpeg-memory-padding 64
   ```
   The option "-mca btl_openib_receive_queues X,4096,1024:X,12288,512:X,65536,512" was added in MPI command because without it there will be error "A process failed to create a queue pair. This usually means either the device has run out of queue pairs (too many connections) or there are insufficient resources available to allocate a queue pair (out of memory). The latter can happen if either 1) insufficient memory is available, or 2) no more physical memory can be registered with the device."
   
   Those environment variables were added because Nvidia set them when creating the Docker container.
   
   I am also trying the code in https://github.com/NVIDIA/DeepLearningExamples/tree/master/MxNet/Classification/RN50v1.5. I noticed this implementation is slightly different than the implementation in MLPerf repo. I will let you know whether it works or not. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services