You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/01/08 09:36:16 UTC
[GitHub] [incubator-mxnet] Justobe opened a new issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Justobe opened a new issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250
I get an exception when I load model using MXNET as keras's backend. **I can load the model correctly with Tensorflow , Theano and cntk as the backend, but I get the following error with MXNET:**
>
> Traceback (most recent call last):
> File "/data/code/lemon-git/scripts/patch_mxnet/crash_checker.py", line 31, in <module>
> model = keras.models.load_model(file_path,custom_objects=ModelUtils.custom_objects())
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 496, in load_model
> model = deserialize_model(f, custom_objects, compile)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 302, in _deserialize_model
> model = model_from_config(model_config, custom_objects=custom_objects)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 535, in model_from_config
> return deserialize(config, custom_objects=custom_objects)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/init.py", line 55, in deserialize
> printable_module_name='layer')
> File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
> list(custom_objects.items())))
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 301, in from_config
> model.add(layer)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 181, in add
> output_tensor = layer(self.outputs[0])
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 470, in call
> output = self.call(inputs, kwargs)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 175, in call
> dilation_rate=self.dilation_rate)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 3705, in conv2d
> padding_mode=padding, data_format=data_format)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 94, in func_wrapper
> train_symbol = func(*args, kwargs)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5045, in _convnd
> padding, is_slice, out_size = _preprocess_padding_mode(padding_mode, x.shape,
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4395, in shape
> return self.get_shape()
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4404, in get_shape
> _, out_shape, _ = self.symbol.infer_shape_partial()
> File "/root/anaconda3/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 1152, in infer_shape_partial
> return self.infer_shape_impl(True, *args, **kwargs)
> File "/root/anaconda3/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 1210, in infer_shape_impl
> ctypes.byref(complete)))
> File "/root/anaconda3/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call
> raise MXNetError(py_str(LIB.MXGetLastError()))
> mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4) :
> Stack trace:
> [bt] (0) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x4b09db) [0x7fd6920089db]
> [bt] (1) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x235e45c) [0x7fd693eb645c]
> [bt] (2) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2620d12) [0x7fd694178d12]
> [bt] (3) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x26235fb) [0x7fd69417b5fb]
> [bt] (4) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShapeEx+0x103e) [0x7fd6940dff7e]
> [bt] (5) /root/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShapePartialEx+0x82) [0x7fd6940e0672]
> [bt] (6) /root/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7fd6c688bec0]
> [bt] (7) /root/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7fd6c688b87d]
> [bt] (8) /root/anaconda3/lib/python3.6/lib-dynload/ctypes.cpython-36m-x86_64-linux-gnu.so(ctypes_callproc+0x2ce) [0x7fd6c6d33ede]
>
The code is as follows:
script.py:
import os
import sys
import argparse
"""Parser of command args"""
parse = argparse.ArgumentParser()
parse.add_argument("--backend", type=str, help="name of backends")
parse.add_argument("--crash_dir", type=str, help="path")
flags, unparsed = parse.parse_known_args(sys.argv[1:])
bk = flags.backend
os.environ['KERAS_BACKEND'] = bk
from keras import backend as K
import keras
import traceback
def custom_objects():
def no_activation(x):
return x
def leakyrelu(x):
import keras.backend as K
return K.relu(x, alpha=0.01)
objects = {}
objects['no_activation'] = no_activation
objects['leakyrelu'] = leakyrelu
return objects
print("INFO:Using {} as backend for states extraction| {} is wanted".format(K.backend(), bk))
files = os.listdir(flags.crash_dir)
pass_cnt = 0
for f in files:
print(f)
file_path = os.path.join(flags.crash_dir,f)
try:
print(f"Loading model. {pass_cnt+1} of {len(files)}")
model = keras.models.load_model(file_path,custom_objects=custom_objects())
pass_cnt += 1
except:
traceback.print_exc()
print(f"Backend:{bk} Total: {len(files)} Pass:{pass_cnt}")
This is a very simple code. Put the model in a folder, and then run the code to load the model with different backend.
You can run like this:(change mxnet to other backend like tensorflow )
> python -u script.py --backend mxnet --crash_dir your/path/
My related library version is:
> numpy 1.16.1 <pip>
> tensorboard 1.14.0 py36hf484d3e_0
> tensorflow-estimator 1.14.0 <pip>
> tensorflow-gpu 1.14.0 <pip>
> Theano 1.0.4
> cntk-gpu 2.7
> mxnet-cu101 1.5.1.post0
> mkl 2018.0.3 1
> mkl-service 1.1.2 py36h17a0993_4
> mkl_fft 1.0.1 py36h3010b51_0
> mkl_random 1.0.1 py36h629b387_0
> Keras 2.2.4 <pip>
> keras-applications 1.0.8 py_0
> keras-mxnet 2.2.4.2 <pip>
> keras-preprocessing 1.1.0 py_1
>
You can download the model from this link:
https://send.firefox.com/download/8798078ff8dc36c7/#yJAfpS_y_69j5WbnJERoPw
thanks in advance!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ptrendx commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
ptrendx commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-572154119
@Justobe Actually, after closer examination of the shape inference of transpose, I believe the current version is correct and should fix your issue. Please try nightly build, which you can find e.g. here: https://repo.mxnet.io/dist/2019-12-30/dist/mxnet_cu101mkl-1.6.0b20191230-py2.py3-none-manylinux1_x86_64.whl
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe edited a comment on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe edited a comment on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-595135552
Sorry for disturbing you again @ptrendx @roywei I find that this model works fine on versions before mxnet1.4.1(includes version 1.4.1). In mxnet 1.5.1, it crashes and throws an exception about the transpose operator. In mxnet 1.6, it still crashes but throws another exception messages like `ValueError: not enough values to unpack (expected 3, got 0)`.
Is it possible that mxnet1.6 fixed the transpose bug but introduced other bugs, or there also exist some other bugs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe edited a comment on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe edited a comment on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-572346111
@ptrendx I really appreciate your reply! I got another exception when I changed the version from 1.5 to 1.6 (both the version you provided and version mxnet-cu101_1.6.0b20191122). I reconfirmed that this exception only occur on the MXNET backend.
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:94: UserWarning: MXNet Backend performs best with `channels_first` format. Using `channels_last` will significantly reduce performance due to the Transpose operations. For performance improvement, please use this API`keras.utils.to_channels_first(x_input)`to transform `channels_last` data to `channels_first` format and also please change the `image_data_format` in `keras.json` to `channels_first`.Note: `x_input` is a Numpy tensor or a list of Numpy tensorRefer to: https://github.com/awslabs/keras-apache-mxnet/tree/master/docs/mxnet_backend/performance_guide.md
> train_symbol = func(*args, **kwargs)
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:97: UserWarning: MXNet Backend performs best with `channels_first` format. Using `channels_last` will significantly reduce performance due to the Transpose operations. For performance improvement, please use this API`keras.utils.to_channels_first(x_input)`to transform `channels_last` data to `channels_first` format and also please change the `image_data_format` in `keras.json` to `channels_first`.Note: `x_input` is a Numpy tensor or a list of Numpy tensorRefer to: https://github.com/awslabs/keras-apache-mxnet/tree/master/docs/mxnet_backend/performance_guide.md
> test_symbol = func(*args, **kwargs)
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:94: UserWarning: MXNet Backend uses `channels_first` format. Axis for BatchNorm should ideally be `1`.Provided - `-1`. Performance can be significantly lower!
> train_symbol = func(*args, **kwargs)
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:97: UserWarning: MXNet Backend uses `channels_first` format. Axis for BatchNorm should ideally be `1`.Provided - `-1`. Performance can be significantly lower!
> test_symbol = func(*args, **kwargs)
> Traceback (most recent call last):
> File "crash_checker.py", line 44, in <module>
> model = keras.models.load_model(file_path,custom_objects=custom_objects())
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 496, in load_model
> model = _deserialize_model(f, custom_objects, compile)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 302, in _deserialize_model
> model = model_from_config(model_config, custom_objects=custom_objects)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 535, in model_from_config
> return deserialize(config, custom_objects=custom_objects)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
> printable_module_name='layer')
> File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
> list(custom_objects.items())))
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 301, in from_config
> model.add(layer)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 181, in add
> output_tensor = layer(self.outputs[0])
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 470, in __call__
> output = self.call(inputs, **kwargs)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 175, in call
> dilation_rate=self.dilation_rate)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 3705, in conv2d
> padding_mode=padding, data_format=data_format)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 94, in func_wrapper
> train_symbol = func(*args, **kwargs)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5047, in _convnd
> filter_dilation)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4870, in _preprocess_padding_mode
> for i in range(nd)])
> ValueError: not enough values to unpack (expected 3, got 0)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-590083123
@ptrendx Thanks for your reply! Hope that mxnet would also support models trained by other backends in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-590068062
> @Justobe Actually, after closer examination of the shape inference of transpose, I believe the current version is correct and should fix your issue. Please try nightly build, which you can find e.g. here: https://repo.mxnet.io/dist/2019-12-30/dist/mxnet_cu101mkl-1.6.0b20191230-py2.py3-none-manylinux1_x86_64.whl
It seems that mxnet has fixed the bug of transpose, but mxnet still does not support models trained by other backends.
Could you share me more information about the bug in transpose? e.g. commit link or PR link .
Looking forward to receiving your reply！ Thanks in advance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ptrendx commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
ptrendx commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-572357202
@roywei ^^
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-572346111
@ptrendx I really appreciate your reply! I got another exception when I change the version from 1.5 to 1.6 (both the version you provided and version mxnet-cu101_1.6.0b20191122). I reconfirmed that this exception only occur on the MXNET backend.
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:94: UserWarning: MXNet Backend performs best with `channels_first` format. Using `channels_last` will significantly reduce performance due to the Transpose operations. For performance improvement, please use this API`keras.utils.to_channels_first(x_input)`to transform `channels_last` data to `channels_first` format and also please change the `image_data_format` in `keras.json` to `channels_first`.Note: `x_input` is a Numpy tensor or a list of Numpy tensorRefer to: https://github.com/awslabs/keras-apache-mxnet/tree/master/docs/mxnet_backend/performance_guide.md
> train_symbol = func(*args, **kwargs)
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:97: UserWarning: MXNet Backend performs best with `channels_first` format. Using `channels_last` will significantly reduce performance due to the Transpose operations. For performance improvement, please use this API`keras.utils.to_channels_first(x_input)`to transform `channels_last` data to `channels_first` format and also please change the `image_data_format` in `keras.json` to `channels_first`.Note: `x_input` is a Numpy tensor or a list of Numpy tensorRefer to: https://github.com/awslabs/keras-apache-mxnet/tree/master/docs/mxnet_backend/performance_guide.md
> test_symbol = func(*args, **kwargs)
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:94: UserWarning: MXNet Backend uses `channels_first` format. Axis for BatchNorm should ideally be `1`.Provided - `-1`. Performance can be significantly lower!
> train_symbol = func(*args, **kwargs)
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:97: UserWarning: MXNet Backend uses `channels_first` format. Axis for BatchNorm should ideally be `1`.Provided - `-1`. Performance can be significantly lower!
> test_symbol = func(*args, **kwargs)
> Traceback (most recent call last):
> File "crash_checker.py", line 44, in <module>
> model = keras.models.load_model(file_path,custom_objects=custom_objects())
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 496, in load_model
> model = _deserialize_model(f, custom_objects, compile)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 302, in _deserialize_model
> model = model_from_config(model_config, custom_objects=custom_objects)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 535, in model_from_config
> return deserialize(config, custom_objects=custom_objects)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
> printable_module_name='layer')
> File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
> list(custom_objects.items())))
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 301, in from_config
> model.add(layer)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 181, in add
> output_tensor = layer(self.outputs[0])
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 470, in __call__
> output = self.call(inputs, **kwargs)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 175, in call
> dilation_rate=self.dilation_rate)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 3705, in conv2d
> padding_mode=padding, data_format=data_format)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 94, in func_wrapper
> train_symbol = func(*args, **kwargs)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5047, in _convnd
> filter_dilation)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4870, in _preprocess_padding_mode
> for i in range(nd)])
> ValueError: not enough values to unpack (expected 3, got 0)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe edited a comment on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe edited a comment on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-572847927
Hello @roywei, thanks for your reply.
1. These models are trained under TensorFlow and can be run correctly under CNTK and Theano. I have a lot of models, and only a few models (including this one) throw exceptions under the MXNET. I wonder if this is a bug in currently mxnet. I'm really looking forward that it could be fix if it's a bug. I enjoy using keras under the mxnet backend (this [Issue17258](https://github.com/apache/incubator-mxnet/issues/17258) may have the same reason)
2. I do notice that you have made a lot of changes in keras-mxnet for mxnet. Will this affect other backends in keras-mxnet?(e.g. TensorFlow, CNTK). This is very exciting if keras-mxnet will not affect other backend and can support mxnet at the same time
3. Besides, I have another [problem](https://github.com/apache/incubator-mxnet/issues/17260).I fail to use inceptionV3 (provided by keras) under mxnet, but it works fine with other backends. It seems that mxnet might have some unexpected behavior when I use API of keras.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-572847927
Hello@roywei, thanks for your reply.
1. These models are trained under TensorFlow and can be run correctly under CNTK and Theano. I have a lot of models, and only a few models (including this one) throw exceptions under the MXNET. I wonder if this is a bug in currently mxnet. I'm really looking forward that it could be fix if it's a bug. I enjoy using keras under the mxnet backend (this [issue17258](https://github.com/apache/incubator-mxnet/issues/17258) may have the same reason)
2. I do notice that you have made a lot of changes in keras-mxnet for mxnet. Will this affect other backends in keras-mxnet?(e.g. TensorFlow, CNTK). This is very exciting if keras-mxnet will not affect other backend and can support mxnet at the same time
3. Besides, I have another [problem](https://github.com/apache/incubator-mxnet/issues/17260).I fail to use inceptionV3 (provided by keras) under mxnet, but it works fine with other backends. It seems that mxnet might have some unexpected behavior when I use API of keras.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe edited a comment on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe edited a comment on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-590068062
> @Justobe Actually, after closer examination of the shape inference of transpose, I believe the current version is correct and should fix your issue. Please try nightly build, which you can find e.g. here: https://repo.mxnet.io/dist/2019-12-30/dist/mxnet_cu101mkl-1.6.0b20191230-py2.py3-none-manylinux1_x86_64.whl
Hi @ptrendx It seems that mxnet has fixed the bug of transpose, but mxnet still does not support models trained by other backends.
Could you share me more information about the bug in transpose? e.g. commit link or PR link .
Looking forward to receiving your reply！ Thanks in advance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe edited a comment on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe edited a comment on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-572346111
@ptrendx I really appreciate your reply! I got another exception when I changed the version from 1.5 to 1.6 (both the version you provided and version mxnet-cu101_1.6.0b20191122). I reconfirmed that this exception will only occur on the MXNET backend.
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:94: UserWarning: MXNet Backend performs best with `channels_first` format. Using `channels_last` will significantly reduce performance due to the Transpose operations. For performance improvement, please use this API`keras.utils.to_channels_first(x_input)`to transform `channels_last` data to `channels_first` format and also please change the `image_data_format` in `keras.json` to `channels_first`.Note: `x_input` is a Numpy tensor or a list of Numpy tensorRefer to: https://github.com/awslabs/keras-apache-mxnet/tree/master/docs/mxnet_backend/performance_guide.md
> train_symbol = func(*args, **kwargs)
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:97: UserWarning: MXNet Backend performs best with `channels_first` format. Using `channels_last` will significantly reduce performance due to the Transpose operations. For performance improvement, please use this API`keras.utils.to_channels_first(x_input)`to transform `channels_last` data to `channels_first` format and also please change the `image_data_format` in `keras.json` to `channels_first`.Note: `x_input` is a Numpy tensor or a list of Numpy tensorRefer to: https://github.com/awslabs/keras-apache-mxnet/tree/master/docs/mxnet_backend/performance_guide.md
> test_symbol = func(*args, **kwargs)
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:94: UserWarning: MXNet Backend uses `channels_first` format. Axis for BatchNorm should ideally be `1`.Provided - `-1`. Performance can be significantly lower!
> train_symbol = func(*args, **kwargs)
> /root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py:97: UserWarning: MXNet Backend uses `channels_first` format. Axis for BatchNorm should ideally be `1`.Provided - `-1`. Performance can be significantly lower!
> test_symbol = func(*args, **kwargs)
> Traceback (most recent call last):
> File "crash_checker.py", line 44, in <module>
> model = keras.models.load_model(file_path,custom_objects=custom_objects())
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 496, in load_model
> model = _deserialize_model(f, custom_objects, compile)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 302, in _deserialize_model
> model = model_from_config(model_config, custom_objects=custom_objects)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 535, in model_from_config
> return deserialize(config, custom_objects=custom_objects)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
> printable_module_name='layer')
> File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
> list(custom_objects.items())))
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 301, in from_config
> model.add(layer)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/sequential.py", line 181, in add
> output_tensor = layer(self.outputs[0])
> File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 470, in __call__
> output = self.call(inputs, **kwargs)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 175, in call
> dilation_rate=self.dilation_rate)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 3705, in conv2d
> padding_mode=padding, data_format=data_format)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 94, in func_wrapper
> train_symbol = func(*args, **kwargs)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 5047, in _convnd
> filter_dilation)
> File "/root/anaconda3/lib/python3.6/site-packages/keras/backend/mxnet_backend.py", line 4870, in _preprocess_padding_mode
> for i in range(nd)])
> ValueError: not enough values to unpack (expected 3, got 0)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ptrendx commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
ptrendx commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-572149516
It is a bug in a shape inference of the Transpose operator. It was changed recently (after 1.5.1 release), but I don't believe your case is covered by those changes (so it would still fail). It should be an easy change, I will make PR to fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] roywei commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
roywei commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-572799442
Hi @Justobe what's the backend used to train your model? Currently keras-mxnet does not support loading model trained using other backends. Mainly due to we made some special cases in Keras front end in order to make it work with mxnet backend. ([ref](https://github.com/awslabs/keras-apache-mxnet/issues/92)). You may have to train your model directly using MNXet backend, save and load with MXNet backend should work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] Justobe commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
Justobe commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-595135552
Hi there @ptrendx @roywei I find that this model works fine on versions before mxnet1.4.1(includes version 1.4.1). In mxnet 1.5.1, it crashes and throws an exception about the transpose operator. In mxnet 1.6, it crashes and throws exception messages like `ValueError: not enough values to unpack (expected 3, got 0)`.
Is it possible that mxnet 1.6 fixed the transpose bug but introduced other bugs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ptrendx commented on issue #17250:
mxnet.base.MXNetError: Error in operator transpose176: [02:14:06]
src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() ==
param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
ptrendx commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-590081950
The PR that fixed the issue in transpose shape inference is #15713.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on issue #17250:
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-616716278
@ptrendx @roywei gentle ping
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] djaym7 commented on issue #17250: mxnet.base.MXNetError: Error in operator transpose176: [02:14:06] src/operator/tensor/./matrix_op-inl.h:354: Check failed: shp.ndim() == param.axes.ndim() (-1 vs. 4)
Posted by GitBox <gi...@apache.org>.
djaym7 commented on issue #17250:
URL: https://github.com/apache/incubator-mxnet/issues/17250#issuecomment-641502090
Getting the same error while using transpose in dataset's transform in dataloader but when transpose is moved outside the transform, the issue is not present
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org