[GitHub] [incubator-mxnet] ann-qin-lu commented on issue #20959: GPU memory leak when using gluon.data.DataLoader with num_workers>0 (with Cuda > 10.1)

[GitBox <gi...@apache.org>]

ann-qin-lu commented on issue #20959:
URL: https://github.com/apache/incubator-mxnet/issues/20959#issuecomment-1067165130


   Some additional resources I've found:
   
   * This is a similar [issue](https://github.com/apache/incubator-mxnet/pull/19924) for CPU memory leak with the MultiWorker setup in DataLoader. The solution was to add the python gc to clean up the memory, however this solution doesn't work for GPU. 
   * The Cudnn release [note](https://docs.nvidia.com/deeplearning/cudnn/release-notes/rel_8.html#rel_8) mentions a new buffer management that might affect the Cuda>=10.2, which seems to be related. And the issue only surfaces after I upgrade Cuda version (tested with Cuda10.2/Cuda11.1/Cuda11.5, and all 3 have memory leak issue).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org