You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/09/19 00:29:24 UTC
[GitHub] [incubator-mxnet] leezu opened a new pull request #19185: Switch to GCC 8 for distribution build
leezu opened a new pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185
## Description ##
Resubmit https://github.com/apache/incubator-mxnet/pull/19034 which was temporarily reverted due to oneDNN issues with GCC 8.
@TaoLV can your team help debug / fix the oneDNN issues?
When both gcc8 + oneDNN 1.6.3 is present, we get the following nan bugs:
```
[2020-09-17T17:48:04.979Z] ______________________ test_dc_hybridblock_deferred_init _______________________
[2020-09-17T17:48:04.979Z] [gw0] linux -- Python 3.6.9 /opt/rh/rh-python36/root/usr/bin/python3
[2020-09-17T17:48:04.979Z]
[2020-09-17T17:48:04.979Z] def test_dc_hybridblock_deferred_init():
[2020-09-17T17:48:04.979Z] class MyBlock(mx.gluon.HybridBlock):
[2020-09-17T17:48:04.979Z] def __init__(self):
[2020-09-17T17:48:04.979Z] super().__init__()
[2020-09-17T17:48:04.979Z] self.dense = mx.gluon.nn.Dense(units=10)
[2020-09-17T17:48:04.979Z] self.weight = mx.gluon.Parameter('weight', allow_deferred_init=True)
[2020-09-17T17:48:04.979Z]
[2020-09-17T17:48:04.979Z] def infer_shape(self, x):
[2020-09-17T17:48:04.979Z] self.weight.shape = (x.shape[1], )
[2020-09-17T17:48:04.979Z]
[2020-09-17T17:48:04.979Z] def forward(self, x):
[2020-09-17T17:48:04.979Z] return self.dense(x) + self.weight.data(x.context)
[2020-09-17T17:48:04.979Z]
[2020-09-17T17:48:04.979Z] net = MyBlock()
[2020-09-17T17:48:04.979Z] net.initialize()
[2020-09-17T17:48:04.979Z] > _assert_dc_gluon(_dc_gluon_simple_setup, net, numpy=False)
[2020-09-17T17:48:04.979Z]
[2020-09-17T17:48:04.979Z] tests/python/unittest/test_deferred_compute.py:504:
[2020-09-17T17:48:04.979Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2020-09-17T17:48:04.979Z] tests/python/unittest/test_deferred_compute.py:421: in _assert_dc_gluon
[2020-09-17T17:48:04.979Z] _all_same(ys_np, ys_hybrid_np)
[2020-09-17T17:48:04.979Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2020-09-17T17:48:04.979Z]
[2020-09-17T17:48:04.979Z] arrays1 = [array([ nan, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] 0.07460972, -0.08127148, -0.32424796,...33878, -0.10624887,
[2020-09-17T17:48:04.979Z] 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] dtype=float32), ...]
[2020-09-17T17:48:04.979Z] arrays2 = [array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] 0.07460972, -0.08127148, -0.32424796,...33878, -0.10624887,
[2020-09-17T17:48:04.979Z] 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] dtype=float32), ...]
[2020-09-17T17:48:04.979Z] message = ''
[2020-09-17T17:48:04.979Z]
[2020-09-17T17:48:04.979Z] def _all_same(arrays1, arrays2, message=''):
[2020-09-17T17:48:04.979Z] same = all(np.array_equal(a1, a2) for a1, a2 in zip(arrays1, arrays2))
[2020-09-17T17:48:04.979Z] if not same:
[2020-09-17T17:48:04.979Z] > raise AssertionError('Arrays not equal ({}):\n{}\n\n{}'.format(message, arrays1, arrays2))
[2020-09-17T17:48:04.979Z] E AssertionError: Arrays not equal ():
[2020-09-17T17:48:04.979Z] E [array([ nan, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ nan, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ nan, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32)]
[2020-09-17T17:48:04.979Z] E
[2020-09-17T17:48:04.979Z] E [array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32), array([ 0.01286458, 0.2107217 , -0.06851891, 0.16233878, -0.10624887,
[2020-09-17T17:48:04.979Z] E 0.07460972, -0.08127148, -0.32424796, -0.0124862 , -0.1862593 ],
[2020-09-17T17:48:04.979Z] E dtype=float32)]
```
Reverting either to gcc7 (which was done) or reverting the oneDNN update (https://github.com/apache/incubator-mxnet/pull/19180) fixes the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-700167847
Thank you @pioy. Can you provide more information about the impact of the bug and the conditions in which it is triggered?
For example, the upcoming 1.8 release includes oneDNN 1.6.3 and presumably is also affected by the bug. In that case we may need to include your fix in the 1.8 release as well, to ensure correctness of mxnet-cpu package which now includes oneDNN by default.
cc @samskalicky
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pioy commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
pioy commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-699009389
Today, on request OneDNN team, I came up with easy reproduction procedure to make easy finding the culprit and fixing it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pioy commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
pioy commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-697995138
I was able to reduce number of tests case in the reproduction sequence to to cases:
```
OMP_NUM_THREADS=1 pytest --maxfail=1 -s -m 'not serial' -n 0 --durations=50 --verbose \
'tests/python/unittest/test_contrib_intgemm.py::test_contrib_intgemm_multiply[api1-16-192-4]'\
tests/python/unittest/test_deferred_compute.py::test_dc_hybridblock_deferred_init
```
Interesting thing is that the introduction of another test in the middle causes the sequence passes:
```
OMP_NUM_THREADS=1 pytest --maxfail=1 -s -m 'not serial' -n 0 --durations=50 --verbose \
'tests/python/unittest/test_contrib_intgemm.py::test_contrib_intgemm_multiply[api1-16-192-4]'\
'tests/python/unittest/test_contrib_intgemm.py::test_contrib_intgemm_multiply[api1-16-192-2]'\
tests/python/unittest/test_deferred_compute.py::test_dc_hybridblock_deferred_init
```
I edited the test test_contrib_intgemm_multiply eliminating calls to test_contrib_intgemm_multiply, what fixed the issue.
It looks like it may be something wrong with contrib_intgemm_fully_connected op.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-700912296
Jenkins CI successfully triggered : [unix-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-700910408
@mxnet-bot run ci [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] DickJC123 commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
DickJC123 commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-696535309
I'm seeing this error too in my PR: See https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-19175/5/pipeline
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pioy commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
pioy commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-699880405
I've got the engineering fix that fixes the issue.
Now we need wait till this fix appears in the onednn repository.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pioy commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
pioy commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-696646681
I have reproduced locally on centos.
Interesting thing is that if I run only this failing test it passes. Also it passes if I run all tests from the test_deferred_compute.py.
The test fails if it runs with all other unit tests.
WIP.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pioy commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
pioy commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-698274033
Hi,
I pinned down the issue to onednn module. I will contact the team to resolve it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-695140157
Here is a screenshot that shows that the build passes when OneDNN update is reverted:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pioy commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
pioy commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-702227141
OneDNN v1.6.4 has been released: https://github.com/oneapi-src/oneDNN/releases/tag/v1.6.4.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pioy commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
pioy commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-698274033
Hi,
I pinned down the issue to onednn module. I will contact the team to resolve it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pioy commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
pioy commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-700603056
The bugs occurs for onednn GEMM calculations with small dimensions (1<n<16).
The bug is triggered by the range of values, that can be interpreted as NaN, in a one zmm registers (zmm24, zmm25, zmm26, or zmm27) just before calling the gemm kernel.
Those values can be remaining of other calculations, likely integer operations (case of this PR).
I assume that float kernels does not return NaN in properly configured pipeline.
The NaN values, if they are not overwritten by other avx512 kernels, may stay long; so they may come from operations that were executed much earlier in the pipeline.
In result of the bug NaNs propagates to the result array. What may terminate execution of operators.
The fix has been merged into master/1.8/1.7. It's ready for customer testing.
See https://github.com/oneapi-src/oneDNN/commit/5ce95efe6f5e86cddbf704b637063cd8dc914125.
There are some other fixes to be merged into 1.6 branch. The tag v1.6.4 will be added after those fixes get merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] TaoLv commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
TaoLv commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-695151515
@leezu Sure, we will take a look at the issue. Do you have steps for us to reproduce it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha merged pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
szha merged pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] pioy commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
pioy commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-696646681
I have reproduced locally on centos.
Interesting thing is that if I run only this failing test it passes. Also it passes if I run all tests from the test_deferred_compute.py.
The test fails if it runs with all other unit tests.
WIP.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] DickJC123 commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
DickJC123 commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-696535309
I'm seeing this error too in my PR: See https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-19175/5/pipeline
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-700912254
@mxnet-bot run ci [unix-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-700910451
Jenkins CI successfully triggered : [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19185: Switch to GCC 8 for distribution build
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19185:
URL: https://github.com/apache/incubator-mxnet/pull/19185#issuecomment-695137389
Hey @leezu , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [windows-cpu, unix-cpu, windows-gpu, website, clang, centos-gpu, unix-gpu, centos-cpu, edge, sanity, miscellaneous]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org