You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/09/15 07:49:09 UTC
[GitHub] [incubator-mxnet] haohuanw opened a new issue #19150: v1.7.x head commit with MKL failure
haohuanw opened a new issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150
## Description
when running with built mxnet-1.7 with head of v1.7.x; if I create a model with even number of channels in batch norm, I got the is_view check failure when using mkldnn.
### Error Message
```
dev-dsk-haohuw-2c-3d588fe6 % MKLDNN_VERBOSE=1 bte python minimum_repro.py
input channel of 45
dnnl_verbose,info,oneDNN v1.6.0 (commit N/A)
dnnl_verbose,info,cpu,runtime:sequential
dnnl_verbose,info,cpu,isa:Intel AVX2
dnnl_verbose,info,gpu,runtime:none
dnnl_verbose,exec,cpu,convolution,gemm:jit,forward_inference,src_f32::blocked:abcde:f0 wei_f32::blocked:abcde:f0 bia_undef::undef::f0 dst_f32::blocked:abcde:f0,,alg:convolution_direct,mb1_ic3oc45_id8od8kd1sd1dd0pd0_ih160oh80kh7sh2dh0ph3_iw160ow80kw7sw2dw0pw3,46.7029
dnnl_verbose,exec,cpu,batch_normalization,ncsp_bnorm:any,forward_inference,data_f32::blocked:abcd:f0 diff_undef::undef::f0,,flags:GS,mb1ic45ih1iw51200,8.98315
input channel of 45, (1, 45, 8, 80, 80)
input channel of 64
dnnl_verbose,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:abcde:f0 dst_f32::blocked:Acdeb8a:f0,,,64x3x1x7x7,0.0170898
dnnl_verbose,exec,cpu,convolution,jit:avx2,forward_inference,src_f32::blocked:abcde:f0 wei_f32::blocked:Acdeb8a:f0 bia_undef::undef::f0 dst_f32::blocked:aBcde8b:f0,,alg:convolution_direct,mb1_ic3oc64_id8od8kd1sd1dd0pd0_ih160oh80kh7sh2dh0ph3_iw160ow80kw7sw2dw0pw3,17.5981
dnnl_verbose,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:abcde:f0 dst_f32::blocked:Acdeb8a:f0,,,64x3x1x7x7,0.0258789
Traceback (most recent call last):
File "minimum_repro.py", line 53, in <module>
out = net(input_data).asnumpy()
File "/home/haohuw/vision-only-workspace/c2h_mainline/env/IhmPoseStateLib-1.0/test-runtime/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2566, in asnumpy
ctypes.c_size_t(data.size)))
File "/home/haohuw/vision-only-workspace/c2h_mainline/env/IhmPoseStateLib-1.0/test-runtime/lib/python3.6/site-packages/mxnet/base.py", line 246, in check_call
raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
File "/home/haohuw/vision-only-workspace/c2h_mainline/src/IhmMXNet/build/private/src/src/ndarray/ndarray.cc", line 650
MXNetError: Check failed: !is_view:
```
## To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
### Steps to reproduce
```
import logging
import os
from mxnet import init
from mxnet.context import cpu
from mxnet.gluon import nn
from mxnet.gluon.block import HybridBlock
from mxnet.gluon.nn import BatchNorm
import mxnet as mx
class BuggyModel(HybridBlock):
def __init__(
self,
channels,
norm_layer=BatchNorm,
norm_kwargs=None,
in_channels=3,
**kwargs
):
super(BuggyModel, self).__init__(**kwargs)
self.in_channels = in_channels
with self.name_scope():
self.conv1 = nn.Conv3D(
in_channels=self.in_channels,
channels=channels,
kernel_size=(1, 7, 7),
strides=(1, 2, 2),
padding=(0, 3, 3),
use_bias=False,
)
self.bn1 = norm_layer(in_channels=channels, **({} if norm_kwargs is None else norm_kwargs))
def hybrid_forward(self, F, x):
"""Hybrid forward of R2+1D net"""
x = self.conv1(x)
x = self.bn1(x)
return x
print(f"input channel of 45")
net = BuggyModel(channels=45)
net.initialize(init=init.Constant(1))
input_data = mx.nd.zeros((1, 3, 8, 160, 160), ctx=mx.cpu())
out = net(input_data).asnumpy()
print(f"input channel of 45, {out.shape}")
print(f"input channel of 64")
net = BuggyModel(channels=64)
net.initialize(init=init.Constant(1))
input_data = mx.nd.zeros((1, 3, 8, 160, 160), ctx=mx.cpu())
out = net(input_data).asnumpy()
print(f"input channel of 64, {out.shape}")
```
## What have you tried to solve it?
1. build with 64f737cdd59fe88d2c5b479f25d011c5156b6a8a solves the issue for now
## Environment
We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:
```
curl --retry 10 -s https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py | python
dev-dsk-haohuw-2c-3d588fe6 % curl --retry 10 -s https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py | python
----------Python Info----------
Version : 3.6.11
Compiler : GCC 7.5.0
Build : ('default', 'Sep 11 2020 22:03:53')
Arch : ('64bit', 'ELF')
------------Pip Info-----------
No corresponding pip install for current python.
----------MXNet Info-----------
No MXNet installed.
----------System Info----------
Platform : Linux-4.9.217-0.1.ac.205.84.332.metal1.x86_64-x86_64-with-redhat-5.3-Tikanga
system : Linux
node : dev-dsk-haohuw-2c-3d588fe6.us-west-2.amazon.com
release : 4.9.217-0.1.ac.205.84.332.metal1.x86_64
version : #1 SMP Thu Apr 2 15:19:24 UTC 2020
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Stepping: 1
CPU MHz: 2699.945
BogoMIPS: 4600.13
Hypervisor vendor: Xen
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 46080K
NUMA node0 CPU(s): 0-7
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0018 sec, LOAD: 0.5492 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0429 sec, LOAD: 0.1730 sec.
Error open Gluon Tutorial(cn): https://zh.gluon.ai, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>, DNS finished in 0.34314703941345215 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0204 sec, LOAD: 0.1066 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0234 sec, LOAD: 0.4174 sec.
Error open Conda: https://repo.continuum.io/pkgs/free/, HTTP Error 403: Forbidden, DNS finished in 0.010480880737304688 sec.
----------Environment----------
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] haohuanw commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
haohuanw commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-693794115
I wasn't got a chance to test other branch; I did couple build using different commit in v1.7.x branch on commit 651e24b281e9e2b405808bd4ec0423a4e6a5a946 and it still fails.
My suspicion is one of:
+ 651e24b281e9e2b405808bd4ec0423a4e6a5a946
+ 3143aabb60038b555db2960d712fb2806b16d581
+ d95de55c322cc67a117958f5ff6faa5d6f25d1b6
introduces the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] akarbown commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
akarbown commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-720449930
Hi @haohuanw, could you please verify the fix on your side?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] pengzhao-intel commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
pengzhao-intel commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-693179623
@anko-intel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] ciyongch commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
ciyongch commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-693909644
Thanks for your inputs, @anko-intel will help to take a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] akarbown commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
akarbown commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-694147981
I've got repro and will look into it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] akarbown commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
akarbown commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-706056225
The issue seems to be present on the other branches as well.
As far as I've checked it has been caused by the [6a5a946](https://github.com/apache/incubator-mxnet/commit/651e24b281e9e2b405808bd4ec0423a4e6a5a946).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] haohuanw commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
haohuanw commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-724498724
sorry I don't really have time on my side to test on the fix but I am also not blocked on this change. feel free to resolve it if we have unit tests that can verify change works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] samskalicky commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
samskalicky commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-724164389
Hi @akarbown I pinged @haohuanw with a reminder. We can close this issue since the fix is in #19299 if we dont hear from him by tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] akarbown commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
akarbown commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-723948242
@samskalicky, could you please tell me who should I ask for verification of that bug in order to close it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] ciyongch commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
ciyongch commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-693777157
Hi @haohuanw Does this failure only happen in v1.7.x branch or the others as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] akarbown commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
akarbown commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-724165192
> Hi @akarbown I pinged @haohuanw with a reminder. We can close this issue since the fix is in #19299 if we dont hear from him by tomorrow.
Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] samskalicky closed issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
samskalicky closed issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] akarbown commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
akarbown commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-706056225
The issue seems to be present on the other branches as well.
As far as I've checked it has been caused by the [6a5a946](https://github.com/apache/incubator-mxnet/commit/651e24b281e9e2b405808bd4ec0423a4e6a5a946).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] samskalicky commented on issue #19150: v1.7.x head commit with MKL failure
Posted by GitBox <gi...@apache.org>.
samskalicky commented on issue #19150:
URL: https://github.com/apache/incubator-mxnet/issues/19150#issuecomment-727101889
Thanks @haohuanw, ill close this issue for now. we can reopen it if you find further issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org