You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by gi...@git.apache.org on 2017/08/11 22:44:17 UTC
[GitHub] haehn commented on issue #3030: Does mxnet support .npy format file?
haehn commented on issue #3030: Does mxnet support .npy format file?
URL: https://github.com/apache/incubator-mxnet/issues/3030#issuecomment-321934681
the same error is thrown if the array is large.. the underlying exception is different tho:
```
print X_train.shape, Y_train.shape, X_train.nbytes
print X_val.shape, Y_val.shape, X_val.nbytes
print X_test.shape, Y_test.shape, X_test.nbytes
(212700, 6, 119, 119) (212700,) 36144536400
(70900, 6, 119, 119) (70900,) 12048178800
(70900, 6, 119, 119) (70900,) 12048178800
#
#
#
t0 = time.time()
batch_size = 100
train_iter = mx.io.NDArrayIter(data=X_train, label=Y_train, batch_size=batch_size)
val_iter = mx.io.NDArrayIter(data=X_val, label=Y_val, batch_size=batch_size)
test_iter = mx.io.NDArrayIter(data=X_test, label=Y_test, batch_size=batch_size)
print 'iterators configured', time.time()-t0, 'seconds'
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-6-713633301713> in <module>()
4 t0 = time.time()
5 batch_size = 100
----> 6 train_iter = mx.io.NDArrayIter(data=X_train, label=Y_train, batch_size=batch_size)
7 val_iter = mx.io.NDArrayIter(data=X_val, label=Y_val, batch_size=batch_size)
8 test_iter = mx.io.NDArrayIter(data=X_test, label=Y_test, batch_size=batch_size)
/home/dhaehn/D1/lib/python2.7/site-packages/mxnet/io.pyc in __init__(self, data, label, batch_size, shuffle, last_batch_handle, data_name, label_name)
577 super(NDArrayIter, self).__init__(batch_size)
578
--> 579 self.data = _init_data(data, allow_empty=False, default_name=data_name)
580 self.label = _init_data(label, allow_empty=True, default_name=label_name)
581
/home/dhaehn/D1/lib/python2.7/site-packages/mxnet/io.pyc in _init_data(data, allow_empty, default_name)
485 except:
486 raise TypeError(("Invalid type '%s' for %s, " % (type(v), k)) + \
--> 487 "should be NDArray or numpy.ndarray")
488
489 return list(data.items())
TypeError: Invalid type '<type 'numpy.ndarray'>' for data, should be NDArray or numpy.ndarray
```
but the real problem is
```
a = mx.nd.array(X_train[0:50000]) # no problem
a = mx.nd.array(X_train[0:60000]) # fails
MXNetError: [16:37:29] include/mxnet/././tensor_blob.h:247: Check failed: this->shape_.Size() == shape.Size() (5097960000 vs. 802992704) TBlob.get_with_shape: new and old shape do not match total elements
Stack trace returned 10 entries:
[bt] (0) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x18b0dc) [0x7f655a9510dc]
[bt] (1) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x27d680) [0x7f655aa43680]
[bt] (2) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x27db65) [0x7f655aa43b65]
[bt] (3) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0xc8047d) [0x7f655b44647d]
[bt] (4) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(+0xc5d29b) [0x7f655b42329b]
[bt] (5) /home/dhaehn/D1/lib/python2.7/site-packages/mxnet/libmxnet.so(MXNDArraySyncCopyFromCPU+0xa) [0x7f655b2f0a1a]
[bt] (6) /lib64/libffi.so.6(ffi_call_unix64+0x4c) [0x7f66065e3dcc]
[bt] (7) /lib64/libffi.so.6(ffi_call+0x1f5) [0x7f66065e36f5]
[bt] (8) /home/dhaehn/D1/lib64/python2.7/lib-dynload/_ctypes.so(_ctypes_callproc+0x30b) [0x7f66067f6c8b]
[bt] (9) /home/dhaehn/D1/lib64/python2.7/lib-dynload/_ctypes.so(+0xaa85) [0x7f66067f0a85]
# but..
a = mx.nd.array(X_train[50000:80000]) # no problem
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services