[GitHub] xu2011 opened a new issue #10793: dataloader load custom dataset error

[GitBox <gi...@apache.org>]

xu2011 opened a new issue #10793: dataloader load custom dataset error
URL: https://github.com/apache/incubator-mxnet/issues/10793
 
 
   Python 3.6 on MacOS
   
   I try to load custom dataset with rec file using dataloader as following:
   `data = gluon.data.RecordFileDataset("face/test.rec")`
   `data_loader = gluon.data.DataLoader(data, batch_size=10, shuffle=True)`
   `for data, label in data_loader:`
   `     do(data)`
       
   ---------------------------------------------------------------------------
   KeyError                                  Traceback (most recent call last)
   <ipython-input-44-e103b90d1e31> in <module>()
         1 data = gluon.data.RecordFileDataset("landmark/test.rec")
         2 data_loader = gluon.data.DataLoader(data, batch_size=10, shuffle=True)
   ----> 3 for data, label in data_loader:
         4     print(label)
         5 
   
   ~/anaconda3/lib/python3.6/site-packages/mxnet/gluon/data/dataloader.py in __iter__(self)
       200         if self._num_workers == 0:
       201             for batch in self._batch_sampler:
   --> 202                 yield self._batchify_fn([self._dataset[idx] for idx in batch])
       203             return
       204 
   
   ~/anaconda3/lib/python3.6/site-packages/mxnet/gluon/data/dataloader.py in default_batchify_fn(data)
        93     else:
        94         data = np.asarray(data)
   ---> 95         return nd.array(data, dtype=data.dtype)
        96 
        97 
   
   ~/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/utils.py in array(source_array, ctx, dtype)
       144         return _sparse_array(source_array, ctx=ctx, dtype=dtype)
       145     else:
   --> 146         return _array(source_array, ctx=ctx, dtype=dtype)
       147 
       148 
   
   ~/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py in array(source_array, ctx, dtype)
      2242             except:
      2243                 raise TypeError('source_array must be array like object')
   -> 2244     arr = empty(source_array.shape, ctx, dtype)
      2245     arr[:] = source_array
      2246     return arr
   
   ~/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py in empty(shape, ctx, dtype)
      3413     if dtype is None:
      3414         dtype = mx_real_t
   -> 3415     return NDArray(handle=_new_alloc_handle(shape, ctx, False, dtype))
   
   ~/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py in _new_alloc_handle(shape, ctx, delay_alloc, dtype)
       136         ctypes.c_int(ctx.device_id),
       137         ctypes.c_int(int(delay_alloc)),
   --> 138         ctypes.c_int(int(_DTYPE_NP_TO_MX[np.dtype(dtype).type])),
       139         ctypes.byref(hdl)))
       140     return hdl
   
   KeyError: <class 'numpy.bytes_'>
   
   ** also, the original dataset don't have labels, I only want to get features with pre-trained models, does anyone know what is right way to do so?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services