You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/07/04 04:20:35 UTC
[GitHub] [incubator-mxnet] maybeLee opened a new issue #20416: MXNetError: Check failed: assign(&dattr, vec.at(i)): Incompatible attr in node at 0-th output: expected [256], got [1]
maybeLee opened a new issue #20416:
URL: https://github.com/apache/incubator-mxnet/issues/20416
## Description
(A clear and concise description of what the bug is.)
I was using one derivation of the AlexNet model to do model inference using MXNet as the Keras's backend. I used the `keras-mxnet` library to equip Keras with the MXNet backend. As a result, MXNet can load the compile this library but report an error while doing model inference. However, when I tried using other three frameworks: TensorFlow, Theano, and CNTK as Keras's backend, I found that these three frameworks can perform the model inference without crash.
### Error Message
(Paste the complete error message. Please also include stack trace by setting environment variable `DMLC_LOG_STACK_TRACE_DEPTH=100` before running your script.)
```
Traceback (most recent call last):
File "get_predictions.py", line 41, in <module>
pred = model.predict(x, batch_size = 16)
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/engine/training.py", line 1184, in predict
steps=steps)
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 295, in predict_loop
batch_outs = f(ins_batch)
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5645, in predict_function
data, label, _, data_shapes, label_shapes = self._adjust_module(inputs, 'pred')
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5525, in _adjust_module
self._set_weights()
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5573, in _set_weights
allow_missing=True)
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/module/bucketing_module.py", line 220, in set_params
force_init=force_init, allow_extra=allow_extra)
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/module/module.py", line 358, in set_params
self._exec_group.set_params(arg_params, aux_params, allow_extra=allow_extra)
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/module/executor_group.py", line 422, in set_params
exec_.copy_params_from(arg_params, aux_params, allow_extra_params=allow_extra)
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/executor.py", line 367, in copy_params_from
array.astype(dst.dtype).copyto(dst)
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2663, in copyto
return _internal._copyto(self, out=other)
File "<string>", line 27, in _copyto
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/_ctypes/ndarray.py", line 91, in _imperative_invoke
ctypes.byref(out_stypes)))
File "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/base.py", line 246, in check_call
raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
[bt] (7) /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeEx+0x62) [0x7f74bc87d6b2]
[bt] (6) /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeImpl(void*, int, void**, int*, void***, int, char const**, char const**)+0x4d7) [0x7f74bc87d127]
[bt] (5) /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&)+0x1c3) [0x7f74bc9da5d3]
[bt] (4) /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, mxnet::DispatchMode*)+0xa1c) [0x7f74bc9e63bc]
[bt] (3) /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/libmxnet.so(bool mxnet::op::ElemwiseShape<1, 1>(nnvm::NodeAttrs const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*)+0x219) [0x7f74bcb94889]
[bt] (2) /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/libmxnet.so(bool mxnet::op::ElemwiseAttrHelper<mxnet::TShape, &mxnet::op::shape_is_none, &mxnet::op::shape_assign, true, &mxnet::op::shape_string, -1, -1>(std::string const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, mxnet::TShape const&)+0x1fc) [0x7f74bcb944bc]
[bt] (1) /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::op::ElemwiseAttrHelper<mxnet::TShape, &mxnet::op::shape_is_none, &mxnet::op::shape_assign, true, &mxnet::op::shape_string, -1, -1>(std::string const&, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, mxnet::TShape const&)::{lambda(std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, unsigned long, char const*)#1}::operator()(std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, unsigned long, char const*) const+0x37a) [0x7f74bcb93c9a]
[bt] (0) /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4f) [0x7f74bc7fb35f]
File "../src/ndarray/./../operator/tensor/../elemwise_op_common.h", line 135
MXNetError: Check failed: assign(&dattr, vec.at(i)): Incompatible attr in node at 0-th output: expected [256], got [1]
```
The model information is as follows:
```
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_1 (Conv2D) (None, 16, 16, 96) 2688
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 8, 8, 96) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 8, 8, 256) 614656
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 3, 3, 256) 0
_________________________________________________________________
batch_normalization_2 (Batch (None, 3, 3, 256) 1024
_________________________________________________________________
conv2d_3 (Conv2D) (None, 3, 3, 384) 885120
_________________________________________________________________
conv2d_4 (Conv2D) (None, 3, 3, 384) 1327488
_________________________________________________________________
conv2d_4 (Conv2D) (None, 3, 3, 384) 1327488
_________________________________________________________________
conv2d_5 (Conv2D) (None, 3, 3, 256) 884992
_________________________________________________________________
softmax_1 (Softmax) (None, 3, 3, 256) 0
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 1, 1, 256) 0
_________________________________________________________________
batch_normalization_3 (Batch (None, 1, 1, 256) 1024
_________________________________________________________________
flatten_1 (Flatten) (None, 256) 0
_________________________________________________________________
dense_1 (Dense) (None, 4096) 1052672
_________________________________________________________________
dropout_1 (Dropout) (None, 4096) 0
_________________________________________________________________
dense_2 (Dense) (None, 4096) 16781312
_________________________________________________________________
dropout_2 (Dropout) (None, 4096) 0
_________________________________________________________________
dense_3 (Dense) (None, 10) 40970
=================================================================
Total params: 22,919,434
Trainable params: 22,918,410
Non-trainable params: 1,024
_________________________________________________________________
```
## To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
You can also access the code in the below zip file.
```
import argparse
import os
import sys
import warnings
parse = argparse.ArgumentParser()
parse.add_argument("-bk", type=str, help="the name of backend")
flags, _ = parse.parse_known_args(sys.argv[1:])
bk = flags.bk
os.environ["KERAS_BACKEND"] = bk
# additional flag to omit some runtime log
os.environ["DMLC_LOG_STACK_TRACE_DEPTH"]="100"
os.environ["MXNET_SUBGRAPH_VERBOSE"]="0"
os.environ["MXNET_CUDNN_AUTOTUNE_DEFAULT"]="0"
os.environ["MXNET_CUDNN_LIB_CHECKING"]="0"
import keras
def custom_objects():
def no_activation(x):
return x
def leakyrelu(x):
import keras.backend as K
return K.relu(x, alpha=0.01)
objects = {}
objects['no_activation'] = no_activation
objects['leakyrelu'] = leakyrelu
return objects
model_name = "model.h5"
warnings.filterwarnings("ignore")
print("start loading model")
model = keras.models.load_model(model_name)
_, (x, _) = keras.datasets.cifar10.load_data()
x = x.astype('float32')/255.0
x = x.reshape(x.shape[0], 32, 32, 3)
x = x[:1]
pred = model.predict(x, batch_size = 16)
print("Successfully get the prediction")
print(pred)
```
### Steps to reproduce
The step to reproduce is very simple:
- Please get the customized model and the reproduced scripts from the zip file [here](https://drive.google.com/drive/folders/1ANXIAboCgBYm-i3ofLQENeCCeODIHFyh?usp=sharing)
- Go to the folder, install the following requirements in your python environment:
```
mxnet-cu101==1.8.0.post0
keras-mxnet==2.2.4.3
h5py==2.10.0
```
- Run the following commands
1. `python get_predictions.py -bk=mxnet`
2. `python get_predictions.py -bk=cntk`
command 1 will lead to a direct crash while command 2 can successfully get the prediction output of the model
## What have you tried to solve it?
1.
2.
## Environment
***We recommend using our script for collecting the diagnostic information with the following command***
`curl --retry 10 -s https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py | python3`
<details>
<summary>Environment Information</summary>
```
----------Python Info----------
Version : 3.6.13
Compiler : GCC 7.5.0
Build : ('default', 'Jun 4 2021 14:25:59')
Arch : ('64bit', '')
------------Pip Info-----------
Version : 21.1.3
Directory : /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version : 1.8.0
Directory : /data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet
Commit hash file "/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/COMMIT_HASH" not found. Not installed from pre-built package or built from source.
Library : ['/data/ziniu/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/mxnet/libmxnet.so']
Build features:
✔ CUDA
✔ CUDNN
✔ NCCL
✔ CUDA_RTC
✖ TENSORRT
✔ CPU_SSE
✔ CPU_SSE2
✔ CPU_SSE3
✖ CPU_SSE4_1
✖ CPU_SSE4_2
✖ CPU_SSE4A
✖ CPU_AVX
✖ CPU_AVX2
✔ OPENMP
✖ SSE
✖ F16C
✖ JEMALLOC
✔ BLAS_OPEN
✖ BLAS_ATLAS
✖ BLAS_MKL
✖ BLAS_APPLE
✔ LAPACK
✔ MKLDNN
✔ OPENCV
✖ CAFFE
✖ PROFILER
✔ DIST_KVSTORE
✖ CXX14
✖ INT64_TENSOR_SIZE
✔ SIGNAL_HANDLER
✖ DEBUG
✖ TVM_OP
----------System Info----------
Platform : Linux-4.18.0-310.el8.x86_64-x86_64-with-centos-8
system : Linux
release : 4.18.0-310.el8.x86_64
version : #1 SMP Tue Jun 8 00:24:50 UTC 2021
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 36
On-line CPU(s) list: 0-35
Thread(s) per core: 2
Core(s) per socket: 18
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz
Stepping: 7
CPU MHz: 3613.195
CPU max MHz: 4800.0000
CPU min MHz: 1200.0000
BogoMIPS: 6000.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0-35
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512_vnni md_clear flush_l1d arch_capabilities
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] maybeLee commented on issue #20416: MXNetError: Check failed: assign(&dattr, vec.at(i)): Incompatible attr in node at 0-th output: expected [256], got [1]
Posted by GitBox <gi...@apache.org>.
maybeLee commented on issue #20416:
URL: https://github.com/apache/incubator-mxnet/issues/20416#issuecomment-877246558
Hi, to make this bug easier to reproduce and understand. I simplify the triggering model into a very easy three-layer randomly generated model.
The model includes three layers: 1 softmax, 1 max pooling, and 1 batch normalization. I create such models using keras and randomly generate weights for batch normalization layer.
You can reproduce the bug by running following code with mxnet version 1.8.0, _you don't need to use all the other trained models:_
```
import os
import argparse
import sys
import warnings
parse = argparse.ArgumentParser()
parse.add_argument("--bk", type=str,default="mxnet", help="the name of backend")
flags, _ = parse.parse_known_args(sys.argv[1:])
os.environ["KERAS_BACKEND"]=flags.bk
import keras
from keras import initializers, layers
import numpy as np
warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings("ignore", category=UserWarning)
model_1 = keras.models.Sequential()
model_1.add(layers.Softmax())
model_1.add(layers.MaxPooling2D())
model_1.add(layers.BatchNormalization())
x = np.random.rand(1,3,3,256)
pred = model_1.predict(x)
print(pred)
```
By running the following command (I assume you save the above toy program as `try.py`:
- `python try --bk mxnet`
You will meet a crash with the same symptom as I mentioned before:
```
Traceback (most recent call last):
File "try.py", line 23, in <module>
pred = model_1.predict(x)
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/engine/training.py", line 1184, in predict
steps=steps)
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 295, in predict_loop
batch_outs = f(ins_batch)
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5645, in predict_function
data, label, _, data_shapes, label_shapes = self._adjust_module(inputs, 'pred')
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5525, in _adjust_module
self._set_weights()
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5573, in _set_weights
allow_missing=True)
File "/mxnet/incubator-mxnet/python/mxnet/module/bucketing_module.py", line 220, in set_params
force_init=force_init, allow_extra=allow_extra)
File "/mxnet/incubator-mxnet/python/mxnet/module/module.py", line 358, in set_params
self._exec_group.set_params(arg_params, aux_params, allow_extra=allow_extra)
File "/mxnet/incubator-mxnet/python/mxnet/module/executor_group.py", line 422, in set_params
exec_.copy_params_from(arg_params, aux_params, allow_extra_params=allow_extra)
File "/mxnet/incubator-mxnet/python/mxnet/executor.py", line 367, in copy_params_from
array.astype(dst.dtype).copyto(dst)
File "/mxnet/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 2663, in copyto
return _internal._copyto(self, out=other)
File "<string>", line 27, in _copyto
File "/mxnet/incubator-mxnet/python/mxnet/_ctypes/ndarray.py", line 91, in _imperative_invoke
ctypes.byref(out_stypes)))
File "/mxnet/incubator-mxnet/python/mxnet/base.py", line 246, in check_call
raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
File "src/operator/numpy/linalg/./../../tensor/../elemwise_op_common.h", line 135
MXNetError: Check failed: assign(&dattr, vec.at(i)): Incompatible attr in node at 0-th output: expected [256], got [1]
```
But if you run such program using CNTK as the backend (`python try.py --bk cntk`), everything works fine.
One interesting thing is that: If I delete either `softmax layer`, `batch normalization` or `max pooling layer`, no crash will happen.
Further, I tried some investigations and guess this is a bug caused by the wrong shape inference of mxnet.
When I change the shape of input to `x=np.random.rand(1, 3, 3, 1)` or `x = np.random.rand(1, 8, 8, 4)` or `x=np.random.rand(1, 5, 3, 1)` , everything works fine and mxnet will not crash.
**But if I set the shape of input to `x=np.random.rand(1,3,3,10)`, which the `-1 th` dimension does not match the `-2 th` dimension after max pooling, mxnet will crash and report such check failed issue.**
Therefore, I assume in some code logic inner elementwise_op_common.h file, it assumes the `-1 th` dimension should be consistent with `-2 th` dimension.
Can you help check whether this is a true problem? And what's is the root cause of such issue?
Indeed thanks for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] maybeLee edited a comment on issue #20416: MXNetError: Check failed: assign(&dattr, vec.at(i)): Incompatible attr in node at 0-th output: expected [256], got [1]
Posted by GitBox <gi...@apache.org>.
maybeLee edited a comment on issue #20416:
URL: https://github.com/apache/incubator-mxnet/issues/20416#issuecomment-877246558
Hi, to make this bug easier to reproduce and understand. I simplify the triggering model into a very easy three-layer randomly generated model.
The model includes three layers: 1 softmax, 1 max pooling, and 1 batch normalization. I create such models using keras and randomly generate weights for batch normalization layer.
You can reproduce the bug by **just running the following program** with mxnet version 1.8.0:
```
import os
import argparse
import sys
import warnings
parse = argparse.ArgumentParser()
parse.add_argument("--bk", type=str,default="mxnet", help="the name of backend")
flags, _ = parse.parse_known_args(sys.argv[1:])
os.environ["KERAS_BACKEND"]=flags.bk
import keras
from keras import initializers, layers
import numpy as np
warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings("ignore", category=UserWarning)
model_1 = keras.models.Sequential()
model_1.add(layers.Softmax())
model_1.add(layers.MaxPooling2D())
model_1.add(layers.BatchNormalization())
x = np.random.rand(1,3,3,256)
pred = model_1.predict(x)
print(pred)
```
By running the following command (I assume you save the above toy program as `try.py`:
- `python try --bk mxnet`
You will meet a crash with the same symptom as I mentioned before:
```
Traceback (most recent call last):
File "try.py", line 23, in <module>
pred = model_1.predict(x)
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/engine/training.py", line 1184, in predict
steps=steps)
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 295, in predict_loop
batch_outs = f(ins_batch)
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5645, in predict_function
data, label, _, data_shapes, label_shapes = self._adjust_module(inputs, 'pred')
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5525, in _adjust_module
self._set_weights()
File "/root/anaconda3/envs/diffcu_mxnet/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5573, in _set_weights
allow_missing=True)
File "/mxnet/incubator-mxnet/python/mxnet/module/bucketing_module.py", line 220, in set_params
force_init=force_init, allow_extra=allow_extra)
File "/mxnet/incubator-mxnet/python/mxnet/module/module.py", line 358, in set_params
self._exec_group.set_params(arg_params, aux_params, allow_extra=allow_extra)
File "/mxnet/incubator-mxnet/python/mxnet/module/executor_group.py", line 422, in set_params
exec_.copy_params_from(arg_params, aux_params, allow_extra_params=allow_extra)
File "/mxnet/incubator-mxnet/python/mxnet/executor.py", line 367, in copy_params_from
array.astype(dst.dtype).copyto(dst)
File "/mxnet/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 2663, in copyto
return _internal._copyto(self, out=other)
File "<string>", line 27, in _copyto
File "/mxnet/incubator-mxnet/python/mxnet/_ctypes/ndarray.py", line 91, in _imperative_invoke
ctypes.byref(out_stypes)))
File "/mxnet/incubator-mxnet/python/mxnet/base.py", line 246, in check_call
raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
File "src/operator/numpy/linalg/./../../tensor/../elemwise_op_common.h", line 135
MXNetError: Check failed: assign(&dattr, vec.at(i)): Incompatible attr in node at 0-th output: expected [256], got [1]
```
But if you run such a program using CNTK as the backend (`python try.py --bk cntk`), everything works fine.
One interesting thing is that: If I delete either `softmax layer`, `batch normalization` or `max pooling layer`, no crash will happen.
Further, I tried some investigations and guess this is a bug caused by the wrong shape inference of mxnet.
When I change the shape of input to `x=np.random.rand(1, 3, 3, 1)` or `x = np.random.rand(1, 8, 8, 4)` or `x=np.random.rand(1, 5, 3, 1)` , everything works fine and mxnet will not crash.
**But if I set the shape of input to `x=np.random.rand(1,3,3,10)`, which the `-1 th` dimension does not match the `-2 th` dimension after max pooling, mxnet will crash and report such check failed issue.**
Therefore, I assume in some code logic inner elementwise_op_common.h file, it assumes the `-1 th` dimension should be consistent with `-2 th` dimension.
Can you help check whether this is a true problem? And what's is the root cause of such issue?
Indeed thanks for your help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] github-actions[bot] commented on issue #20416: MXNetError: Check failed: assign(&dattr, vec.at(i)): Incompatible attr in node at 0-th output: expected [256], got [1]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #20416:
URL: https://github.com/apache/incubator-mxnet/issues/20416#issuecomment-873518859
Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue.
Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly.
If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on [contributing to MXNet](https://mxnet.apache.org/community/contribute) and our [development guides wiki](https://cwiki.apache.org/confluence/display/MXNET/Developments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org