You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/09/13 03:35:35 UTC
[GitHub] WenmuZhou edited a comment on issue #9974: DataLoader with workers
not compatible with ImageRecordDataset
WenmuZhou edited a comment on issue #9974: DataLoader with workers not compatible with ImageRecordDataset
URL: https://github.com/apache/incubator-mxnet/issues/9974#issuecomment-420872512
I still have this problem in 1.3.0
```sh
/data1/zj/crnn.gluon/venv/bin/python /data1/zj/crnn.gluon/dataset.py
50761
0.055738210678100586
0.08428478240966797
0.07413840293884277
0.05549335479736328
0.08482122421264648
0.0804448127746582
0.057969093322753906
0.08210444450378418
0.06818866729736328
0.05565834045410156
0.0641944408416748
0.05271577835083008
0.07879519462585449
0.05999612808227539
0.06945180892944336
0.037737369537353516
0.06135749816894531
0.07474946975708008
0.06603407859802246
0.04885411262512207
0.06480669975280762
0.06882929801940918
0.07541894912719727
0.0850071907043457
0.04295015335083008
0.0550844669342041
0.0662848949432373
0.05972623825073242
0.07331728935241699
0.061258792877197266
0.05453753471374512
Traceback (most recent call last):
File "/data1/zj/crnn.gluon/dataset.py", line 148, in <module>
for i, (img, label) in enumerate(data_loader):
File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/dataloader.py", line 347, in same_process_iter
ret = self._batchify_fn([self._dataset[idx] for idx in batch])
File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/dataloader.py", line 347, in <listcomp>
ret = self._batchify_fn([self._dataset[idx] for idx in batch])
File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/dataset.py", line 131, in __getitem__
item = self._data[idx]
File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/vision/datasets.py", line 257, in __getitem__
record = super(ImageRecordDataset, self).__getitem__(idx)
File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/gluon/data/dataset.py", line 189, in __getitem__
return self._record.read_idx(self._record.keys[idx])
File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/recordio.py", line 264, in read_idx
self.seek(idx)
File "/data1/zj/crnn.gluon/venv/lib/python3.5/site-packages/mxnet/recordio.py", line 227, in seek
check_call(_LIB.MXRecordIOReaderSeek(self.handle, pos))
KeyboardInterrupt
Process finished with exit code 1
```
the code is
```python
from mxnet.gluon.data import DataLoader
from mxnet.gluon.data.vision.datasets import ImageRecordDataset
from mxnet.gluon.data.vision.transforms import ToTensor
dataset = ImageRecordDataset('/data1/zj/data/crnn/txt/val.rec')
data_loader = DataLoader(dataset.transform_first(ToTensor()), 1, shuffle=True, num_workers=6)
print(len(dataset))
start = time.time()
for i, (img, label) in enumerate(data_loader):
if (i + 1) % 10 == 0:
print(time.time() - start)
start = time.time()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services