You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2017/12/27 08:25:01 UTC
[GitHub] mwbyeon commented on issue #9207: It seems nd.array or NDArrayIter not support too large array
mwbyeon commented on issue #9207: It seems nd.array or NDArrayIter not support too large array
URL: https://github.com/apache/incubator-mxnet/issues/9207#issuecomment-354074516
@fcbruce
It works.
```
$ pip install mxnet
$ python
Python 3.6.3 (default, Oct 19 2017, 23:50:38)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mxnet as mx
>>> mx.__version__
'1.0.0'
>>> import numpy as np
>>> X = np.zeros((20000, 32768), dtype=np.float32)
>>> mx.nd.array(X)
[[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
...,
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]
[ 0. 0. 0. ..., 0. 0. 0.]]
<NDArray 20000x32768 @cpu(0)>
>>>
```
what's your memory capacity?
your array is required 2.4GB memory (20000*32768*4bytes).
(I tested on MacOS with 16GB Memory)
if exceed memory capacity, it occurs above error message.
```
>>> mx.nd.array(np.zeros((200000, 32768), dtype=np.float32))
[17:23:34] /Users/travis/build/dmlc/mxnet-distro/mxnet-build/dmlc-core/include/dmlc/logging.h:308: [17:23:34] include/mxnet/./tensor_blob.h:276: Check failed: this->shape_.Size() == shape.Size() (6553600000 vs. 2258632704) TBlob.get_with_shape: new and old shape do not match total elements
Stack trace returned 7 entries:
[bt] (0) 0 libmxnet.so 0x0000000107116b98 _ZN4dmlc15LogMessageFatalD2Ev + 40
[bt] (1) 1 libmxnet.so 0x000000010713b7c9 _ZNK5mxnet5TBlob14get_with_shapeIN7mshadow3cpuELi1EfEENS2_6TensorIT_XT0_ET1_EERKNS2_5ShapeIXT0_EEEPNS2_6StreamIS5_EE + 777
[bt] (2) 2 libmxnet.so 0x000000010819427e _ZN5mxnet7ndarray4CopyIN7mshadow3cpuES3_EEvRKNS_5TBlobEPS4_NS_7ContextES8_NS_10RunContextE + 14382
[bt] (3) 3 libmxnet.so 0x000000010816e1e5 _ZNK5mxnet7NDArray15SyncCopyFromCPUEPKvm + 1109
[bt] (4) 4 libmxnet.so 0x0000000107ffb2cd MXNDArraySyncCopyFromCPU + 13
[bt] (5) 5 _ctypes.cpython-36m-darwin.so 0x000000010628d02f ffi_call_unix64 + 79
[bt] (6) 6 ??? 0x00007ffeea4422a0 0x0 + 140732828754592
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/dylan/.pyenv/versions/mxnet/lib/python3.6/site-packages/mxnet/ndarray/utils.py", line 146, in array
return _array(source_array, ctx=ctx, dtype=dtype)
File "/Users/dylan/.pyenv/versions/mxnet/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2245, in array
arr[:] = source_array
File "/Users/dylan/.pyenv/versions/mxnet/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 437, in __setitem__
self._set_nd_basic_indexing(key, value)
File "/Users/dylan/.pyenv/versions/mxnet/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 698, in _set_nd_basic_indexing
self._sync_copyfrom(value)
File "/Users/dylan/.pyenv/versions/mxnet/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 863, in _sync_copyfrom
ctypes.c_size_t(source_array.size)))
File "/Users/dylan/.pyenv/versions/mxnet/lib/python3.6/site-packages/mxnet/base.py", line 146, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [17:23:34] include/mxnet/./tensor_blob.h:276: Check failed: this->shape_.Size() == shape.Size() (6553600000 vs. 2258632704) TBlob.get_with_shape: new and old shape do not match total elements
Stack trace returned 7 entries:
[bt] (0) 0 libmxnet.so 0x0000000107116b98 _ZN4dmlc15LogMessageFatalD2Ev + 40
[bt] (1) 1 libmxnet.so 0x000000010713b7c9 _ZNK5mxnet5TBlob14get_with_shapeIN7mshadow3cpuELi1EfEENS2_6TensorIT_XT0_ET1_EERKNS2_5ShapeIXT0_EEEPNS2_6StreamIS5_EE + 777
[bt] (2) 2 libmxnet.so 0x000000010819427e _ZN5mxnet7ndarray4CopyIN7mshadow3cpuES3_EEvRKNS_5TBlobEPS4_NS_7ContextES8_NS_10RunContextE + 14382
[bt] (3) 3 libmxnet.so 0x000000010816e1e5 _ZNK5mxnet7NDArray15SyncCopyFromCPUEPKvm + 1109
[bt] (4) 4 libmxnet.so 0x0000000107ffb2cd MXNDArraySyncCopyFromCPU + 13
[bt] (5) 5 _ctypes.cpython-36m-darwin.so 0x000000010628d02f ffi_call_unix64 + 79
[bt] (6) 6 ??? 0x00007ffeea4422a0 0x0 + 140732828754592
>>>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services