[GitHub] ArmageddonKnight opened a new issue #11402: Understanding Memory Allocation in MXNet

[GitBox <gi...@apache.org>]

ArmageddonKnight opened a new issue #11402: Understanding Memory Allocation in MXNet
URL: https://github.com/apache/incubator-mxnet/issues/11402
 
 
   Hi there,
   
   I have a question regarding the memory allocation in MXNet. I am trying to obtain the memory **breakdown** of the RNN language modeling example located in `example/rnn/word_lm`. However, after a naive implementation of putting a trace after the `cudaMalloc` call located in `src/storage/pooled_storage_manager.h` and `src/storage/gpu_device_storage.h`, like the following:
   
   ```C++
     cudaError_t e = cudaMalloc(&ret, size);
   
     // Trace the cudaMalloc function.
     static double accumulated_alloc_mem = 0.0;
   
     std::cout << "cudaMalloc from file " << __FILE__ << " is called." << std::endl;
     std::cout << "\t" "Allocated Memory (MB): " << size * 1.0 / 1e6 << std::endl;
     std::cout << "\t" "Allocated Memory (Accumulated, MB): " <<
       (accumulated_alloc_mem += size * 1.0 / 1e6) << std::endl;
     std::cout << "\t" << dmlc::StackTrace() << std::endl;
   
     ...
   ```
   
   I noticed that this is only able to capture around `200` MB out of the total allocated `700` MB memory. 
   
   Therefore, my question is the following:
   
   - What is file `gpu_device_storage.h` used for? I see that it is neither included in any other files nor compiled.
   - What are other possible places for GPU memory allocation to happen?
   
   Thanks for the help.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services