You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2022/03/17 17:26:00 UTC
[GitHub] [incubator-mxnet] DickJC123 opened a new issue #20967: Unittest failures seen in test_dnnl.py
DickJC123 opened a new issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967
## Description
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/2/pipeline/ (test_conv_subgraph.py::test_pos_conv_act_add[True-gelu-True-data_shape1])
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/3/pipeline/288 (test_conv_subgraph.py::test_pos_conv_act_add[True-leakyrelu-True-data_shape1])
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/4/pipeline/289 (test_fc_subgraph.py::test_fc_transpose[mxnet.ndarray-auto-False-data_shape2])
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1073966623
Hi @DickJC123,
1. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]
- Not related to oneDNN upgrade, @anko-intel analyzed it and here is PR with the fix https://github.com/apache/incubator-mxnet/pull/20969
2. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/2/pipeline/ (test_conv_subgraph.py::test_pos_conv_act_add[True-gelu-True-data_shape1])
AND
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/3/pipeline/288 (test_conv_subgraph.py::test_pos_conv_act_add[True-leakyrelu-True-data_shape1])
- These two suffer from reading data from a neither updated nor zeroed register during convolution with the number of input channels lower than 4 and blocked weights (https://github.com/apache/incubator-mxnet/issues/20826). The full fix will arrive in v2.6 and temporarily we will force plain format (at least in this axis) to make the convolution to work properly and the test to pass PR: https://github.com/apache/incubator-mxnet/pull/20970.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 edited a comment on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
DickJC123 edited a comment on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1071357448
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]
The unix-cpu Python3:oneDNN-CPU and Python3:oneDNN-MKL-CPU jobs together seem to have a very low passing rate (1 in 5 runs passed by my experience in PR-20965).
Not sure if this is new behavior, but if so, could it be related to the recent onednn upgrade @bartekkuncer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1076452266
I do not know about xdist, but will look into these tests and try to speed them up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 commented on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
DickJC123 commented on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1071357448
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]
The unix-cpu Python3:oneDNN-CPU and Python3:oneDNN-MKL-CPU jobs together seem to have a very low passing rate (1 in 5 runs passed by my experience in PR-20965).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bartekkuncer edited a comment on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
bartekkuncer edited a comment on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1073966623
Hi @DickJC123,
1. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]
- Not related to oneDNN upgrade, @anko-intel analyzed it and here is PR with the fix https://github.com/apache/incubator-mxnet/pull/20969
2. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/2/pipeline/ (test_conv_subgraph.py::test_pos_conv_act_add[True-gelu-True-data_shape1])
AND
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/3/pipeline/288 (test_conv_subgraph.py::test_pos_conv_act_add[True-leakyrelu-True-data_shape1])
- These two suffer from reading data from a neither updated nor zeroed register during convolution with the number of input channels lower than 4 and blocked weights (https://github.com/apache/incubator-mxnet/issues/20826). The full fix will arrive in v2.6 of oneDNN and temporarily we will force plain format (at least in this axis) to make the convolution work properly and the test to pass PR: https://github.com/apache/incubator-mxnet/pull/20970.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1072223282
Hi @DickJC123, I will look into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bartekkuncer edited a comment on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
bartekkuncer edited a comment on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1073966623
Hi @DickJC123,
1. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]
- Not related to oneDNN upgrade, @anko-intel analyzed it and here is PR with the fix https://github.com/apache/incubator-mxnet/pull/20969
2. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/2/pipeline/ (test_conv_subgraph.py::test_pos_conv_act_add[True-gelu-True-data_shape1])
AND
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/3/pipeline/288 (test_conv_subgraph.py::test_pos_conv_act_add[True-leakyrelu-True-data_shape1])
- These two suffer from reading data from a neither updated nor zeroed register during convolution with the number of input channels lower than 4 and blocked weights (https://github.com/apache/incubator-mxnet/issues/20826). The full fix will arrive in v2.6 of oneDNN, therefore temporarily we will force plain format (at least on this axis) to make the convolution work properly and the test to pass PR: https://github.com/apache/incubator-mxnet/pull/20970.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bartekkuncer edited a comment on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
bartekkuncer edited a comment on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1073966623
Hi @DickJC123,
1. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]
- Not related to oneDNN upgrade, @anko-intel analyzed it and here is PR with the fix https://github.com/apache/incubator-mxnet/pull/20969
2. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/2/pipeline/ (test_conv_subgraph.py::test_pos_conv_act_add[True-gelu-True-data_shape1])
AND
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/3/pipeline/288 (test_conv_subgraph.py::test_pos_conv_act_add[True-leakyrelu-True-data_shape1])
- These two suffer from reading data from a neither updated nor zeroed register during convolution with the number of input channels lower than 4 and blocked weights (https://github.com/apache/incubator-mxnet/issues/20826). The full fix will arrive in v2.6 of oneDNN, therefore temporarily we will force plain format (at least in this axis) to make the convolution work properly and the test to pass PR: https://github.com/apache/incubator-mxnet/pull/20970.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 edited a comment on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
DickJC123 edited a comment on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1071357448
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]
The unix-cpu Python3:oneDNN-CPU and Python3:oneDNN-MKL-CPU jobs together seem to have a very low passing rate (1 in 5 runs passed by my experience in PR-20965).
Not sure if this is new behavior, but if so, could it be related to the recent onednn upgrade @bartekkuncer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 commented on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
DickJC123 commented on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1071357448
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]
The unix-cpu Python3:oneDNN-CPU and Python3:oneDNN-MKL-CPU jobs together seem to have a very low passing rate (1 in 5 runs passed by my experience in PR-20965).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bartekkuncer edited a comment on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
bartekkuncer edited a comment on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1073966623
Hi @DickJC123,
1. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/5/pipeline/288/ test_fc_subgraph.py::test_fc_transpose[mxnet.numpy-int8-True-data_shape2]
- Not related to oneDNN upgrade, @anko-intel analyzed it and here is PR with the fix https://github.com/apache/incubator-mxnet/pull/20969
2. https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/2/pipeline/ (test_conv_subgraph.py::test_pos_conv_act_add[True-gelu-True-data_shape1])
AND
https://jenkins.mxnet-ci.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20965/3/pipeline/288 (test_conv_subgraph.py::test_pos_conv_act_add[True-leakyrelu-True-data_shape1])
- These two suffer from reading data from a neither updated nor zeroed register during convolution with the number of input channels lower than 4 and blocked weights (https://github.com/apache/incubator-mxnet/issues/20826). The full fix will arrive in v2.6 and temporarily we will force plain format (at least in this axis) to make the convolution work properly and the test to pass PR: https://github.com/apache/incubator-mxnet/pull/20970.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 commented on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
DickJC123 commented on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1074161900
Thank you very much for this thorough response! I'll look forward to smoother CI pipelines runs once these fixes are merged.
On an unrelated point, I notice the dnnl tests push out the complete Python3-onednn-cpu job runtime to 2hr, 15min. For example, there are these tests:
```
[2022-03-17T23:13:35.468Z] 900.98s call tests/python/dnnl/test_quantization_dnnl.py::test_quantize_gluon_with_forward
[2022-03-17T23:13:35.468Z] 404.96s call tests/python/dnnl/test_dnnl.py::test_Deconvolution
[2022-03-17T23:13:35.468Z] 232.17s call tests/python/dnnl/test_dnnl.py::test_convolution
[2022-03-17T23:13:35.468Z] 220.28s call tests/python/dnnl/test_dnnl.py::test_pooling
```
Is it possible to trim down the runtime of these tests, or perhaps run the tests with xdist on multiple workers (each with fewer cores presumably)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1072223282
Hi @DickJC123, I will look into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bartekkuncer commented on issue #20967: Unittest failures seen in test_dnnl
Posted by GitBox <gi...@apache.org>.
bartekkuncer commented on issue #20967:
URL: https://github.com/apache/incubator-mxnet/issues/20967#issuecomment-1076164394
@anko-intel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org