[GitHub] [incubator-mxnet] szha opened a new pull request #16194: retry GPU memory allocation if fragmented

[GitBox <gi...@apache.org>]

szha opened a new pull request #16194: retry GPU memory allocation if fragmented
URL: https://github.com/apache/incubator-mxnet/pull/16194
 
 
   ## Description ##
   retry GPU memory allocation if allocation failure occurs due to fragmentation.
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [x] Changes are complete (i.e. I finished coding on this PR)
   - [x] All changes have test coverage:
   - [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [x] retry GPU memory allocation if allocation failure occurs due to fragmentation.
   
   ## Comments ##
   - Memory fragmentation happens at runtime when total free memory is larger than the requested memory chunk, but no consecutive space can be found to accommodate the memory request.
   - I'm changing the memory pool allocation logic when cudaErrorMemoryAllocation occurrs:
     1. release all cached unused memory chunks and retry 
     2. retry memory allocation

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services