[GitHub] [incubator-mxnet] drmasud opened a new issue #16960: large numpy array to mxnet ndarray conversion

[GitBox <gi...@apache.org>]

drmasud opened a new issue #16960: large numpy array to mxnet ndarray conversion
URL: https://github.com/apache/incubator-mxnet/issues/16960
 
 
   ## Description
   Conversion fails on large matrices.
   
   ### Error Message
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/utils.py", line 146, in array
       return _array(source_array, ctx=ctx, dtype=dtype)
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2505, in array
       arr[:] = source_array
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 449, in __setitem__
       self._set_nd_basic_indexing(key, value)
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 715, in _set_nd_basic_indexing
       self._sync_copyfrom(value)
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 881, in _sync_copyfrom
       ctypes.c_size_t(source_array.size)))
     File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [19:09:04] src/ndarray/ndarray_function.cc:51: Check failed: size == to->Size() (-294967296 vs. 4000000000) : copying size mismatch, from: 18446744072529682432 bytes, to: 16000000000 bytes.
   Stack trace:
     [bt] (0) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2795cb) [0x7efff4d2c5cb]
     [bt] (1) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x259399b) [0x7efff704699b]
     [bt] (2) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::NDArray::SyncCopyFromCPU(void const*, unsigned long) const+0x284) [0x7efff6fe5934]
     [bt] (3) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXNDArraySyncCopyFromCPU+0x2b) [0x7efff6d8ebcb]
     [bt] (4) /usr/lib64/libffi.so.6(ffi_call_unix64+0x4c) [0x7f0059bbacec]
     [bt] (5) /usr/lib64/libffi.so.6(ffi_call+0x1f5) [0x7f0059bba615]
     [bt] (6) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2a0) [0x7f0059dcd290]
     [bt] (7) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x9586) [0x7f0059dc6586]
     [bt] (8) /usr/lib64/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x90) [0x7f0061bce7e0]
   
   
   ### Steps to reproduce
   T= nd.array(np.random.randn(5000000,800))
   (Paste the commands you ran that produced the error.)
   
   ## What have you tried to solve it?
   
   1. workaround: If you convert in smaller chunks and concatenate the ndarray to create the final mxnet
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services