[GitHub] [incubator-mxnet] eric-haibin-lin commented on a change in pull request #16114: improve dataloader signals and messages

[GitBox <gi...@apache.org>]

eric-haibin-lin commented on a change in pull request #16114: improve dataloader signals and messages
URL: https://github.com/apache/incubator-mxnet/pull/16114#discussion_r323511316
 
 

 ##########
 File path: python/mxnet/gluon/data/dataloader.py
 ##########
 @@ -537,16 +557,21 @@ def default_batchify_fn(data):
         If ``True``, use threading pool instead of multiprocessing pool. Using threadpool
         can avoid shared memory usage. If `DataLoader` is more IO bounded or GIL is not a killing
         problem, threadpool version may achieve better performance than multiprocessing.
-
+    timeout : int, default is 120
+        The timeout in seconds for each worker to fetch a batch data. Only modify this number
+        unless you are experiencing timeout and you know it's due to slow data loading.
+        Sometimes full `shared_memory` will cause all workers to hang and causes timeout. In these
+        cases please reduce `num_workers` or increase system `shared_memory` size instead.
     """
     def __init__(self, dataset, batch_size=None, shuffle=False, sampler=None,
                  last_batch=None, batch_sampler=None, batchify_fn=None,
                  num_workers=0, pin_memory=False, pin_device_id=0,
-                 prefetch=None, thread_pool=False):
+                 prefetch=None, thread_pool=False, timeout=120):
         self._dataset = dataset
         self._pin_memory = pin_memory
         self._pin_device_id = pin_device_id
         self._thread_pool = thread_pool
+        self._timeout = timeout
 
 Review comment:
   add a check for positive values for timeout? Otherwise lgtm 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services