[GitHub] [incubator-mxnet] chandana1332 commented on issue #15025: Gluon DataLoader incorrectly terminates the process pool in 1.4

[GitBox <gi...@apache.org>]

chandana1332 commented on issue #15025: Gluon DataLoader incorrectly terminates the process pool in 1.4
URL: https://github.com/apache/incubator-mxnet/issues/15025#issuecomment-496558157
 
 
   >>The worker pool is managed by dataloader for sake of resource conservation in case users created hundreds of iterators out of the same dataloader.One way to fix this problem is to use ref counting for how many iterators are using the worker pool.
   
   Got it! I see why the worker pool init was moved inside DataLoader. Ref counting is a good option but it might be better if we let python handle scope of objects rather than managing it on our side. 
   One way to let python handle it is to pass a reference of DataLoader to the iterator. This way DataLoader will not go out of scope untill all the iterators have gone out of scope (example: https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html#DataLoader)
   What do you think?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services