[GitHub] [incubator-mxnet] CanyonWind edited a comment on issue #16424: [Channel Shuffle / Hard Swish / Hard Sigmoid] running in MKL CPU backend failed

[GitBox <gi...@apache.org>]

CanyonWind edited a comment on issue #16424: [Channel Shuffle / Hard Swish / Hard Sigmoid] running in MKL CPU backend failed
URL: https://github.com/apache/incubator-mxnet/issues/16424#issuecomment-551277963
 
 
   Hi @ZhennanQin, thanks a lot for your effort! 
   I tried to verify the quantized model's performance with the nightly built release (`mxnet-mkl-1.6.0b20191107`, the merge commit was completed on 1106 so I assumed this release already contains the updated codes) on the Mac to get a quick result. 
   
   ```sh
   git clone https://github.com/CanyonWind/Single-Path-One-Shot-NAS-MXNet.git
   cd Single-Path-One-Shot-NAS-MXNet
   python3 -m venv env
   source env/bin/activate
   pip install mxnet-mkl --pre
   cd quantization
   ```
   I've tried the following:
   With calib-mode: `none`
   ```sh
   # quantize model
   python3 quantize_mkldnn.py --model=ShuffleNas_fixArch --num-calib-batches=5 --calib-mode=none
   # verify performance
   python3 imagenet_inference.py --symbol-file model/ShuffleNas_fixArch-quantized-symbol.json --param-file model/ShuffleNas_fixArch-quantized-0000.params --rgb-mean=123.68,116.779,103.939 --rgb-std=58.393,57.12,57.375 --num-skipped-batches=50 --batch-size=64 --num-inference-batches=5 --dataset=./data/val_256_q90.rec --ctx=cpu
   # accuracy: 0.009375
   ```
   With calib-mode: `naive`
   ```sh
   # quantize model
   python quantize_mkldnn.py --model=ShuffleNas_fixArch --num-calib-batches=5 --calib-mode=naive
   # verify performance
   python3 imagenet_inference.py --symbol-file model/ShuffleNas_fixArch-quantized-5batches-naive-symbol.json --param-file model/ShuffleNas_fixArch-quantized-0000.params --rgb-mean=123.68,116.779,103.939 --rgb-std=58.393,57.12,57.375 --num-skipped-batches=50 --batch-size=64 --num-inference-batches=5 --dataset=./data/val_256_q90.rec --ctx=cpu
   # accuracy: 0.003125
   ```
   With calib-mode: `entropy`
   ```sh
   # quantize model
   python3 quantize_mkldnn.py --model=ShuffleNas_fixArch --num-calib-batches=5 --calib-mode=entropy
   # verify performance
   python3 imagenet_inference.py --symbol-file model/ShuffleNas_fixArch-quantized-5batches-entropy-symbol.json --param-file model/ShuffleNas_fixArch-quantized-0000.params --rgb-mean=123.68,116.779,103.939 --rgb-std=58.393,57.12,57.375 --num-skipped-batches=50 --batch-size=64 --num-inference-batches=5 --dataset=./data/val_256_q90.rec --ctx=cpu
   # error was thrown when doing inference
   ```
   Could you please guide me how did you verify the quantization accuracy or could you please try any of the above quantization procedure (it wouldn't take more than 10min to finish) at your convenience? 
   
   Thanks again for your generous help, I do appreciate it a lot!
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services