[GitHub] YutingZhang commented on a change in pull request #13318: Improving multi-processing reliability for gluon DataLoader

[GitBox <gi...@apache.org>]

YutingZhang commented on a change in pull request #13318: Improving multi-processing reliability for gluon DataLoader
URL: https://github.com/apache/incubator-mxnet/pull/13318#discussion_r235264331
 
 

 ##########
 File path: python/mxnet/gluon/data/dataloader.py
 ##########
 @@ -275,9 +284,24 @@ def __iter__(self):
     def shutdown(self):
         """Shutdown internal workers by pushing terminate signals."""
         if not self._shutdown:
+            # send shutdown signal to the fetcher and join data queue first
+            # Remark:   loop_fetcher need to be joined prior to the workers.
+            #           otherwise, the the fetcher may fail at getting data
+            self._data_queue.put((None, None))
+            self._fetcher.join()
+            # send shutdown signal to all worker processes
             for _ in range(self._num_workers):
                 self._key_queue.put((None, None))
-            self._data_queue.put((None, None))
+            # wait 0.5 sec for workers to shut down if necessary
+            for w in self._workers:
+                if w.is_alive():
+                    import time
+                    time.sleep(0.5)   # allow 0.5 sec to join
 
 Review comment:
   I have two options:
   1) simply remove this loop. Then, do not wait the workers to finish, and just kill them if they are alive. 
   2) fork a new process to do the waiting and termination.  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services