You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2022/09/05 13:09:56 UTC
[GitHub] [incubator-mxnet] anko-intel opened a new pull request, #21134: [DOC] Add custom strategy script to quantization with INC example
anko-intel opened a new pull request, #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134
## Description ##
Add script with custom strategy for Intel® Neural Compressor.
Update results to measured on latest MxNet build
(2.0.0b20220902 (commit 3a19f0e50d75fedb05eb558a9c835726b57df4cf))
Results:
## Checklist ##
### Essentials ###
- [ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
- [ ] Changes are complete (i.e. I finished coding on this PR)
- [ ] All changes have test coverage
- [ ] Code is well-documented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21134: [DOC] Add custom strategy script to quantization with INC example
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134#issuecomment-1238984057
Jenkins CI successfully triggered : [unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] bgawrych merged pull request #21134: [DOC] Add custom strategy script to quantization with INC example
Posted by GitBox <gi...@apache.org>.
bgawrych merged PR #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21134: [DOC] Add custom strategy script to quantization with INC example
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134#issuecomment-1239415276
Jenkins CI successfully triggered : [centos-gpu, unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] anko-intel commented on pull request #21134: [DOC] Add custom strategy script to quantization with INC example
Posted by GitBox <gi...@apache.org>.
anko-intel commented on PR #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134#issuecomment-1239415132
@mxnet-bot run ci [centos-gpu, unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21134: [DOC] Add custom strategy script to quantization with INC example
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134#issuecomment-1237667096
Jenkins CI successfully triggered : [unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] anko-intel commented on pull request #21134: [DOC] Add custom strategy script to quantization with INC example
Posted by GitBox <gi...@apache.org>.
anko-intel commented on PR #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134#issuecomment-1238983980
@mxnet-bot run ci [unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #21134: [DOC] Add custom strategy script to quantization with INC example
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on PR #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134#issuecomment-1237001611
Hey @anko-intel , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [clang, unix-cpu, windows-gpu, unix-gpu, edge, windows-cpu, website, centos-gpu, miscellaneous, centos-cpu, sanity]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] anko-intel commented on a diff in pull request #21134: [DOC] Add custom strategy script to quantization with INC example
Posted by GitBox <gi...@apache.org>.
anko-intel commented on code in PR #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134#discussion_r964650620
##########
docs/python_docs/python/tutorials/performance/backend/dnnl/dnnl_quantization_inc.md:
##########
@@ -244,34 +244,42 @@ It is done by
[resnet_tuning.py](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/resnet_tuning.py)
script on a small part of data set to reduce time required for tuning (9 batches).
Later saved models are validated on a whole data set by
-[resnet_measurment.py](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/resnet_measurment.py)
+[resnet_measurement.py](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/resnet_measurement.py)
script.
Accuracy results on the whole validation dataset (782 batches) are shown below.
| Optimization method | Top 1 accuracy | Top 5 accuracy | Top 1 relative accuracy loss [%] | Top 5 relative accuracy loss [%] | Cost = one-time optimization on 9 batches [s] | Validation time [s] | Speedup |
-|----------------------|-------:|-------:|-----:|-----:|-------:|--------:|-----:|
-| fp32 no optimization | 0.7699 | 0.9340 | 0.0 | 0.0 | 0.00 | 1448.69 | 1.0 |
-| fp32 fused | 0.7699 | 0.9340 | 99.9 | 99.5 | 0.03 | 149.45 | 9.7 |
-| int8 full naive | 0.2207 | 0.3912 | 71.3 | 58.1 | 12.74 | 46.28 | 31.3 |
-| int8 full entropy | 0.6933 | 0.8917 | 9.9 | 4.5 | 81.50 | 47.07 | 30.8 |
-| int8 smart naive | 0.2210 | 0.3905 | 71.3 | 58.2 | 12.55 | 46.56 | 31.1 |
-| int8 smart entropy | 0.6928 | 0.8910 | 10.0 | 4.6 | 80.89 | 46.58 | 31.1 |
-| int8 INC basic | 0.7692 | 0.9331 | 0.1 | 0.1 | 526.47 | 48.68 | 29.8 |
-| int8 INC mse | 0.7692 | 0.9337 | **0.1** | **0.0** | 227.89 | 50.19 | **28.9** |
+|----------------------|-------:|-------:|------:|------:|-------:|--------:|------:|
+| fp32 no optimization 0.7699 | 0.9340 | 0.00 | 0.00 | 0.00 | 316.50 | 1.0 |
+| fp32 fused 0.7699 | 0.9340 | 0.00 | 0.00 | 0.03 | 147.77 | 2.1 |
+| int8 full naive 0.2207 | 0.3912 | 71.33 | 58.12 | 11.29 | 45.81 | **6.9** |
+| int8 full entropy 0.6933 | 0.8917 | 9.95 | 4.53 | 80.23 | 46.39 | 6.8 |
+| int8 smart naive 0.2210 | 0.3905 | 71.29 | 58.19 | 11.15 | 46.02 | 6.9 |
+| int8 smart entropy 0.6928 | 0.8910 | 10.01 | 4.60 | 79.75 | 45.98 | 6.9 |
+| int8 INC basic 0.7692 | 0.9331 | **0.09** | 0.10 | 266.50 | 48.32 | **6.6** |
+| int8 INC mse 0.7692 | 0.9337 | **0.09** | 0.03 | 106.50 | 49.76 | **6.4** |
+| int8 INC mycustom 0.7699 | 0.9338 | **0.00** | 0.02 | 370.29 | 70.07 | **4.5** |
+
Environment:
- Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz (c6i.16xlarge Amazon EC2 instance)
- Ubuntu 20.04.4 LTS (GNU/Linux Ubuntu 20.04.4 LTS 5.15.0-1017-aws ami-0558cee5b20db1f9c)
-- MXNet 2.0.0b20220823 (commit daac02c7854ffa71bc11fd950c2d6c9ea356b394 )
+- MXNet 2.0.0b20220902 (commit 3a19f0e50d75fedb05eb558a9c835726b57df4cf)
- INC 1.13.1
- scripts above were run as parameter for [run.sh](https://github.com/apache/incubator-mxnet/blob/master/benchmark/python/dnnl/run.sh)
script to properly setup parallel computation parameters.
-For this model INC basic and mse strategies found configurations meeting the 1.5% relative accuracy
+For this model INC `basic`, `mse` and `mycustom` strategies found configurations meeting the 1.5% relative accuracy
loss criterion. Only the `bayesian` strategy didn't find solution within 500 attempts limit.
Although these results may suggest that the `mse` strategy is the best compromise between time spent
to find the optimized model and final model performance efficiency, different strategies may give
-better results for specific models and tasks. You can notice, that the most important thing done by INC
+better results for specific models and tasks. For example for ALBERT model there is no solution
+given by build-in INC strategies. For such situation you can create your custom strategy, similar
+to this one:
+[custom_strategy.py](https://github.com/apache/incubator-mxnet/blob/master/example/quantization_inc/custom_strategy.py).
+You can find description of this particular custom strategy in the Medium article
+(add link to article).
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] anko-intel commented on pull request #21134: [DOC] Add custom strategy script to quantization with INC example
Posted by GitBox <gi...@apache.org>.
anko-intel commented on PR #21134:
URL: https://github.com/apache/incubator-mxnet/pull/21134#issuecomment-1237667046
@mxnet-bot run ci [unix-gpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org