You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/02/27 05:30:22 UTC

[GitHub] [incubator-mxnet] eric-haibin-lin opened a new issue #17705: mkldnn quantized FC is slow

eric-haibin-lin opened a new issue #17705: mkldnn quantized FC is slow
URL: https://github.com/apache/incubator-mxnet/issues/17705
 
 
   quantized BERT model with int8 is 2x slower than float32. 
   
   Download trained SST params from: https://dist-bert.s3.amazonaws.com/demo/finetune/sst.params 
   
   Clone is install gluon-nlp v0.9
   ```
   # calibration
   KMP_AFFINITY=granularity=fine,noduplicates,compact,1,0 OMP_NUM_THREADS=1 numactl --physcpubind=0 --membind=0 python3 finetune_classifier.py --task_name SST --only_calibration --model_parameters  sst.params
   # fp32 inference
   python3 finetune_classifier.py --task_name SST --epoch 1 --only_inference --model_parameters sst.params --round_to 128 --dev_batch_size 1
   # int8 inference
   python3 finetune_classifier.py --task_name SST --epoch 1 --only_inference --model_prefix ./output_dir/model_bert_SST_quantized_customize --deploy --round_to 128 --dev_batch_size 1
   ```
   
   I'm using c5.12xlarge. I tried to set OMP_NUM_THREAD=8, but int8 is still slower than float32. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] pengzhao-intel commented on issue #17705: mkldnn quantized FC is slow

Posted by GitBox <gi...@apache.org>.
pengzhao-intel commented on issue #17705: mkldnn quantized FC is slow
URL: https://github.com/apache/incubator-mxnet/issues/17705#issuecomment-591789973
 
 
   @wuxun-zhang @ciyongch 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] ciyongch commented on issue #17705: mkldnn quantized FC is slow

Posted by GitBox <gi...@apache.org>.
ciyongch commented on issue #17705: mkldnn quantized FC is slow
URL: https://github.com/apache/incubator-mxnet/issues/17705#issuecomment-591791543
 
 
   Thanks for reporting this, I will take a look.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] ciyongch commented on issue #17705: mkldnn quantized FC is slow

Posted by GitBox <gi...@apache.org>.
ciyongch commented on issue #17705: mkldnn quantized FC is slow
URL: https://github.com/apache/incubator-mxnet/issues/17705#issuecomment-591868495
 
 
   @eric-haibin-lin I just created a PR https://github.com/apache/incubator-mxnet/pull/17707 to address this issue, please take a review.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] pengzhao-intel commented on issue #17705: mkldnn quantized FC is slow

Posted by GitBox <gi...@apache.org>.
pengzhao-intel commented on issue #17705: mkldnn quantized FC is slow
URL: https://github.com/apache/incubator-mxnet/issues/17705#issuecomment-597046840
 
 
   Feel free to reopen if the issue is not resolved.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] pengzhao-intel closed issue #17705: mkldnn quantized FC is slow

Posted by GitBox <gi...@apache.org>.
pengzhao-intel closed issue #17705: mkldnn quantized FC is slow
URL: https://github.com/apache/incubator-mxnet/issues/17705
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services