[GitHub] ashkanaev opened a new issue #11421: Can't iterate custom dataset by dataloader with 'num_workers > 1' param on Linux

[GitBox <gi...@apache.org>]

ashkanaev opened a new issue #11421: Can't iterate custom dataset by dataloader with 'num_workers > 1' param on Linux
URL: https://github.com/apache/incubator-mxnet/issues/11421
 
 
   Hi!
   
   Let me introduce some problem which I've got runing this code:
   
   First of all I have custom DataSet based on Dataset (mxnet.gluon.data)
   
   ```
   class CustomDataSet(Dataset):
       def __init__(self, root):
           self.root = os.path.join(root)
           self.img_paths = []
           
           self._img = os.path.join(root, '{}.png')
           
           for fn in os.listdir(os.path.join(root)):
               if len(fn) > 3 and fn[-4:] == '.png':
                   self.img_paths.append(fn[:-4])
           
       def __len__(self):
           return len(self.img_paths)
       
       def __getitem__(self, idx):
           img_path = self._img.format(self.img_paths[idx])
   
           img = cv2.imread(img_path)
           img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
           img = mx.nd.array(img)
           return img
   ```
   I define dataset and loader like this:
   
   ```
       train_dataset = MultiBranchDataSet(args.dataset, 'train', ctx, args.class_weight, args.ignored_class, args.outputs)
       train_loader = DataLoader(train_dataset, batch_size=args.batch_size, shuffle=False, last_batch='rollover', num_workers = 4)
   ```
   And trying to complete this code:
   
   ```
   for data in train_loader:
   	print(data.shape)
   ```
   
   It's working on windows with num_workers > 1, but does't work on Linux on the same parametr.
   
   >   File "/home/user/kpavel/mxnet_27_06/python/mxnet/gluon/data/dataloader.py", line 218, in __next__
   >     idx, batch = self._data_queue.get()
   >   File "/home/user/anaconda3/envs/mx/lib/python3.5/multiprocessing/queues.py", line 335, in get
   >     res = self._reader.recv_bytes()
   >   File "/home/user/anaconda3/envs/mx/lib/python3.5/multiprocessing/connection.py", line 216, in recv_bytes
   >     buf = self._recv_bytes(maxlength)
   >   File "/home/user/anaconda3/envs/mx/lib/python3.5/multiprocessing/connection.py", line 407, in _recv_bytes
   >     buf = self._recv(4)
   >   File "/home/user/anaconda3/envs/mx/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
   >     chunk = read(handle, remaining)
   
   Another words, I can't use Custom dataset and iterate my data in multiple threads.
   
   Have you any ideas?
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services