[GitHub] [incubator-mxnet] bgawrych commented on issue #21076: NDArrays leak their memory when deleted

[GitBox <gi...@apache.org>]

bgawrych commented on issue #21076:
URL: https://github.com/apache/incubator-mxnet/issues/21076#issuecomment-1178702865

   Hi @matteosal, it is expected behaviour as MXNet by default uses Pooled StorageManager. So memory buffer is cached and can be reused for other array. See code snippet:
   ```
   import mxnet as mx
   import os, psutil
   
   print('branch: ' + mx.runtime.get_branch())
   print('commit: ' + mx.runtime.get_commit_hash())
   
   def get_memory_usage():
           return psutil.Process(os.getpid()).memory_info().rss / 1e+6
   
   print("before alloc:", get_memory_usage())
   nd = mx.nd.ones([1000000, 1000])
   mx.ndarray.waitall()
   print("after alloc: ", get_memory_usage())
   del(nd)
   mx.ndarray.waitall()
   print("after delete: ", get_memory_usage())
   
   
   nd2 = mx.nd.ones([1000000, 1000])
   mx.ndarray.waitall()
   print("after nd2 alloc: ", get_memory_usage())
   ```
   
   
   Output:
   ```
   branch: HEAD
   commit: e36c9f075aa7ee14e1a8ccf245df5d1e1648515b
   before alloc: 178.159616
   [08:20:11] ../src/storage/storage.cc:202: Using Pooled (Naive) StorageManager for CPU
   after alloc:  4174.946304
   after delete:  4174.946304
   after nd2 alloc:  4174.946304
   ```
   
   But with `del` statement commented:
   ```
   branch: HEAD
   commit: e36c9f075aa7ee14e1a8ccf245df5d1e1648515b
   before alloc: 177.11104
   [08:19:03] ../src/storage/storage.cc:202: Using Pooled (Naive) StorageManager for CPU
   after alloc:  4174.024704
   after delete:  4174.024704
   after nd2 alloc:  8176.889856
   ``` 
   
   If you don't want to use this feature you can change to unpooled storage engine by env flag: `MXNET_CPU_MEM_POOL_TYPE=Unpooled`
   
   https://mxnet.apache.org/versions/master/api/faq/env_var
   
   Let me know if we can close this issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org