[GitHub] [incubator-mxnet] samskalicky opened a new issue #16996: mx.io.NDArrayIter cant pad when size is large

[GitBox <gi...@apache.org>]

samskalicky opened a new issue #16996: mx.io.NDArrayIter cant pad when size is large
URL: https://github.com/apache/incubator-mxnet/issues/16996
 
 
   ## Description
   When input data is smaller than batch size, sometime it errors out with:
   ```
   Traceback (most recent call last):
     File "test.py", line 7, in <module>
       for batch in dataiter:
     File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 230, in __next__
       return self.next()
     File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 682, in next
       data = self.getdata()
     File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 764, in getdata
       return self._batchify(self.data)
     File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 751, in _batchify
       second_data = self._getdata(data_source, end=pad)
     File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 707, in _getdata
       ]]) for x in data_source
     File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 707, in <listcomp>
       ]]) for x in data_source
     File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 705, in __getitem__
       return self._slice(key.start, key.stop)
     File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 1338, in _slice
       start, stop, _ = _get_index_range(start, stop, self.shape[0])
     File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 3080, in _get_index_range
       raise IndexError('Slicing stop %d exceeds limit of %d' % (stop, length))
   IndexError: Slicing stop 190 exceeds limit of 10
   ```
   
   ## To Reproduce
   ```
   import numpy as np
   import mxnet as mx
   
   data = np.arange(40).reshape((10,2,2))
   dataiter = mx.io.NDArrayIter(data=data, batch_size=200, last_batch_handle='pad')
   for batch in dataiter:
        print(batch.data[0].asnumpy().shape)
   ```
   
   Using a batch size of 20 succeeds, but larger sizes fail. 
   
   ### Steps to reproduce
   This is using the 1.6.0 branch
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services