[GitHub] WenmuZhou commented on issue #9974: DataLoader with workers not compatible with ImageRecordDataset

[GitBox <gi...@apache.org>]

WenmuZhou commented on issue #9974: DataLoader with workers not compatible with ImageRecordDataset
URL: https://github.com/apache/incubator-mxnet/issues/9974#issuecomment-420872512
 
 
   I still have this problem in 1.3.0
   ```sh
   /data1/zj/crnn.gluon/venv/bin/python /data1/zj/crnn.gluon/dataset.py
   50761
   0.055738210678100586
   0.08428478240966797
   0.07413840293884277
   0.05549335479736328
   0.08482122421264648
   0.0804448127746582
   0.057969093322753906
   0.08210444450378418
   0.06818866729736328
   0.05565834045410156
   0.0641944408416748
   0.05271577835083008
   0.07879519462585449
   0.05999612808227539
   0.06945180892944336
   0.037737369537353516
   0.06135749816894531
   0.07474946975708008
   0.06603407859802246
   0.04885411262512207
   0.06480669975280762
   0.06882929801940918
   0.07541894912719727
   0.0850071907043457
   0.04295015335083008
   0.0550844669342041
   0.0662848949432373
   0.05972623825073242
   0.07331728935241699
   0.061258792877197266
   0.05453753471374512
   Traceback (most recent call last):
     File "/data1/zj/crnn.gluon/dataset.py", line 148, in <module>
       for i, (img, label) in enumerate(data_loader):
     File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/dataloader.py", line 347, in same_process_iter
       ret = self._batchify_fn([self._dataset[idx] for idx in batch])
     File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/dataloader.py", line 347, in <listcomp>
       ret = self._batchify_fn([self._dataset[idx] for idx in batch])
     File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/dataset.py", line 131, in __getitem__
       item = self._data[idx]
     File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/vision/datasets.py", line 257, in __getitem__
       record = super(ImageRecordDataset, self).__getitem__(idx)
     File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/dataset.py", line 189, in __getitem__
       return self._record.read_idx(self._record.keys[idx])
     File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/recordio.py", line 264, in read_idx
       self.seek(idx)
     File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/recordio.py", line 227, in seek
       check_call(_LIB.MXRecordIOReaderSeek(self.handle, pos))
   KeyboardInterrupt
   
   Process finished with exit code 1
   ```
   the code is 
   ```python
       from mxnet.gluon.data import DataLoader
       from mxnet.gluon.data.vision.datasets import ImageRecordDataset
   
       from mxnet.gluon.data.vision.transforms import ToTensor
   
       dataset = ImageRecordDataset('/data1/zj/data/crnn/txt/val.rec')
       data_loader = DataLoader(dataset.transform_first(ToTensor()), 1, shuffle=True, num_workers=0)
       print(len(dataset))
       start = time.time()
       for i, (img, label) in enumerate(data_loader):
           if (i + 1) % 10 == 0:
               print(time.time() - start)
               start = time.time()
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services