You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/12/19 17:56:58 UTC

[GitHub] [incubator-mxnet] lorenzob opened a new issue #17126: Memory should be completely released after an OOM happens

lorenzob opened a new issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126
 
 
   ## Description
   After a cudaMalloc failed: out of memory error is raised everything becomes unusable with more out of memory errors (even if a smaller batch is provided).
   
   If I load two models both become unusable after the OOM.
   
   ### Error Message
   Traceback (most recent call last):
     File "core/facerec.py", line 673, in <module>
       img_result, distances = compare(list(images_data), "work/")
     File "core/facerec.py", line 183, in compare
       embedding = model.model.get_outputs()[0].asnumpy()
     File "/home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 1996, in asnumpy
       ctypes.c_size_t(data.size)))
     File "/home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [18:53:03] src/storage/./pooled_storage_manager.h:157: cudaMalloc failed: out of memory
   Stack trace:
     [bt] (0) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x4b04cb) [0x7f1c3a28f4cb]
     [bt] (1) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e653eb) [0x7f1c3cc443eb]
     [bt] (2) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e6af0f) [0x7f1c3cc49f0f]
     [bt] (3) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::NDArray::CheckAndAlloc() const+0x1cc) [0x7f1c3a307aac]
     [bt] (4) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25db3b8) [0x7f1c3c3ba3b8]
     [bt] (5) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25db777) [0x7f1c3c3ba777]
     [bt] (6) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25e1b6d) [0x7f1c3c3c0b6d]
     [bt] (7) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c1cd1) [0x7f1c3c3a0cd1]
     [bt] (8) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25c51e0) [0x7f1c3c3a41e0]
   
   ## To Reproduce
   
   Invoke the model with a large enough batch to get an OOM. Now call it again with a small batch that would not throw as OOM if called on its own. This small batch throws an OOM too.
   
   It looks like the memory from the previous call is not completely released.
   
   ## What have you tried to solve it?
   
   mxnet.context.current_context().empty_cache()
   mxnet.gpu(0).empty_cache()
   
   ## Environment
   
   We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:
   
   ----------Python Info----------
   Version      : 3.6.7
   Compiler     : GCC 7.3.0
   Build        : ('default', 'Oct 23 2018 19:16:44')
   Arch         : ('64bit', '')
   ------------Pip Info-----------
   Version      : 19.1
   Directory    : /home/trz/miniconda3/envs/CenterNet/lib/python3.6/site-packages/pip
   ----------MXNet Info-----------
   Version      : 1.5.1
   Directory    : /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet
   Num GPUs     : 1
   Commit Hash   : c9818480680f84daa6e281a974ab263691302ba8
   ----------System Info----------
   Platform     : Linux-4.15.0-72-generic-x86_64-with-debian-buster-sid
   system       : Linux
   node         : acqua
   release      : 4.15.0-72-generic
   version      : #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019
   ----------Hardware Info----------
   machine      : x86_64
   processor    : x86_64
   Architecture:        x86_64
   CPU op-mode(s):      32-bit, 64-bit
   Byte Order:          Little Endian
   CPU(s):              12
   On-line CPU(s) list: 0-11
   Thread(s) per core:  2
   Core(s) per socket:  6
   Socket(s):           1
   NUMA node(s):        1
   Vendor ID:           GenuineIntel
   CPU family:          6
   Model:               158
   Model name:          Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
   Stepping:            10
   CPU MHz:             3790.389
   CPU max MHz:         4100,0000
   CPU min MHz:         800,0000
   BogoMIPS:            4416.00
   Virtualization:      VT-x
   L1d cache:           32K
   L1i cache:           32K
   L2 cache:            256K
   L3 cache:            9216K
   NUMA node0 CPU(s):   0-11
   Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d
   ----------Network Test----------
   Setting timeout: 10
   Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0279 sec, LOAD: 0.5680 sec.
   Timing for GluonNLP GitHub: https://github.com/dmlc/gluon-nlp, DNS: 0.0017 sec, LOAD: 0.5182 sec.
   Timing for GluonNLP: http://gluon-nlp.mxnet.io, DNS: 0.0592 sec, LOAD: 0.5766 sec.
   Timing for D2L: http://d2l.ai, DNS: 0.0408 sec, LOAD: 0.0685 sec.
   Timing for D2L (zh-cn): http://zh.d2l.ai, DNS: 0.0288 sec, LOAD: 0.2421 sec.
   Timing for FashionMNIST: https://repo.mxnet.io/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0983 sec, LOAD: 0.8585 sec.
   Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0438 sec, LOAD: 0.5219 sec.
   Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0302 sec, LOAD: 0.1939 sec.
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-567874505
 
 
   @leezu I'm willing too but I'm stuck with the build on a gtest error and I cannot find any complete build guide covering all the requisites. Can you help me with this?
   
   I have gtest in /usr/src/gtest and /usr/local/lib/gtest/
   
   I tried:
   
   cmake -DBLAS=open -DUSE_CUDA=1 -DUSE_CUDA_PATH=/usr/local/cuda -DUSE_CUDNN=1 -DUSE_MKL_IF_AVAILABLE=ON -DGTEST_ROOT=/usr/local/lib/gtest/ -DCMAKE_BUILD_TYPE=Release -GNinja ..
   
   but I have this error:
   ```
   [CMakeError.log](https://github.com/apache/incubator-mxnet/files/3987836/CMakeError.log)
   [CMakeOutput.log](https://github.com/apache/incubator-mxnet/files/3987837/CMakeOutput.log)
   [CMakeError.log](https://github.com/apache/incubator-mxnet/files/3987838/CMakeError.log)
   [CMakeOutput.log](https://github.com/apache/incubator-mxnet/files/3987839/CMakeOutput.log)
   
   
   
   
   -- CMAKE_CROSSCOMPILING FALSE
   -- CMAKE_HOST_SYSTEM_PROCESSOR x86_64
   -- CMAKE_SYSTEM_PROCESSOR x86_64
   -- CMAKE_SYSTEM_NAME Linux
   -- CMake version '3.10.2' using generator 'Ninja'
   -- Determining F16C support
   -- F16C enabled
   CMake Error at CMakeLists.txt:281 (add_subdirectory):
     The source directory
   
       /home/trz/github/incubator-mxnet/3rdparty/mkldnn
   
     does not contain a CMakeLists.txt file.
   
   
   -- Found NVTX (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libnvToolsExt.so)
   -- Found MKL (include: /opt/intel/mkl/include, lib: /opt/intel/mkl/lib/intel64/libmkl_rt.so
   -- CUDA detected: 10.0
   -- Found cuDNN (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
   -- Running GPU architecture autodetection
   -- Found CUDA arch 6.1
   -- Added CUDA NVCC flags for: sm_61
   -- Using JEMalloc malloc
   -- OpenCV 3.2.0 found (/usr/share/OpenCV)
   --  OpenCV_LIBS=opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs
   USE_LAPACK is ON
   CMake Error at CMakeLists.txt:516 (add_subdirectory):
     add_subdirectory given source
     "/home/trz/github/incubator-mxnet/3rdparty/googletest/googletest" which is
     not an existing directory.
   
   
   CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
   Please set them or make sure they are set and tested correctly in the CMake files:
   CUDA_cublas_device_LIBRARY (ADVANCED)
       linked by target "im2rec" in directory /home/trz/github/incubator-mxnet
       linked by target "im2rec" in directory /home/trz/github/incubator-mxnet
       linked by target "mxnet" in directory /home/trz/github/incubator-mxnet
       linked by target "mxnet" in directory /home/trz/github/incubator-mxnet
       linked by target "mxnet_unit_tests" in directory /home/trz/github/incubator-mxnet/tests
       linked by target "mxnet_unit_tests" in directory /home/trz/github/incubator-mxnet/tests
       linked by target "image-classification-predict" in directory /home/trz/github/incubator-mxnet/example/image-classification/predict-cpp
       linked by target "image-classification-predict" in directory /home/trz/github/incubator-mxnet/example/image-classification/predict-cpp
   
   -- Configuring incomplete, errors occurred!
   See also "/home/trz/github/incubator-mxnet/build/CMakeFiles/CMakeOutput.log".
   See also "/home/trz/github/incubator-mxnet/build/CMakeFiles/CMakeError.log".
   ```
   
   [CMakeError.log](https://github.com/apache/incubator-mxnet/files/3987841/CMakeError.log)
   [CMakeOutput.log](https://github.com/apache/incubator-mxnet/files/3987842/CMakeOutput.log)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-567874505
 
 
   @leezu I'm willing too but I'm stuck with the build on a gtest error and I cannot find any complete build guide covering all the requisites. Can you help me with this?
   
   I have gtest in /usr/src/gtest and /usr/local/lib/gtest/
   
   I tried:
   
   cmake -DBLAS=open -DUSE_CUDA=1 -DUSE_CUDA_PATH=/usr/local/cuda -DUSE_CUDNN=1 -DUSE_MKL_IF_AVAILABLE=ON -DGTEST_ROOT=/usr/local/lib/gtest/ -DCMAKE_BUILD_TYPE=Release -GNinja ..
   
   but I have this error:
   ```
   [CMakeError.log](https://github.com/apache/incubator-mxnet/files/3987836/CMakeError.log)
   [CMakeOutput.log](https://github.com/apache/incubator-mxnet/files/3987837/CMakeOutput.log)
   [CMakeError.log](https://github.com/apache/incubator-mxnet/files/3987838/CMakeError.log)
   [CMakeOutput.log](https://github.com/apache/incubator-mxnet/files/3987839/CMakeOutput.log)
   
   
   
   
   -- CMAKE_CROSSCOMPILING FALSE
   -- CMAKE_HOST_SYSTEM_PROCESSOR x86_64
   -- CMAKE_SYSTEM_PROCESSOR x86_64
   -- CMAKE_SYSTEM_NAME Linux
   -- CMake version '3.10.2' using generator 'Ninja'
   -- Determining F16C support
   -- F16C enabled
   CMake Error at CMakeLists.txt:281 (add_subdirectory):
     The source directory
   
       /home/trz/github/incubator-mxnet/3rdparty/mkldnn
   
     does not contain a CMakeLists.txt file.
   
   
   -- Found NVTX (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libnvToolsExt.so)
   -- Found MKL (include: /opt/intel/mkl/include, lib: /opt/intel/mkl/lib/intel64/libmkl_rt.so
   -- CUDA detected: 10.0
   -- Found cuDNN (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
   -- Running GPU architecture autodetection
   -- Found CUDA arch 6.1
   -- Added CUDA NVCC flags for: sm_61
   -- Using JEMalloc malloc
   -- OpenCV 3.2.0 found (/usr/share/OpenCV)
   --  OpenCV_LIBS=opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs
   USE_LAPACK is ON
   CMake Error at CMakeLists.txt:516 (add_subdirectory):
     add_subdirectory given source
     "/home/trz/github/incubator-mxnet/3rdparty/googletest/googletest" which is
     not an existing directory.
   
   
   CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
   Please set them or make sure they are set and tested correctly in the CMake files:
   CUDA_cublas_device_LIBRARY (ADVANCED)
       linked by target "im2rec" in directory /home/trz/github/incubator-mxnet
       linked by target "im2rec" in directory /home/trz/github/incubator-mxnet
       linked by target "mxnet" in directory /home/trz/github/incubator-mxnet
       linked by target "mxnet" in directory /home/trz/github/incubator-mxnet
       linked by target "mxnet_unit_tests" in directory /home/trz/github/incubator-mxnet/tests
       linked by target "mxnet_unit_tests" in directory /home/trz/github/incubator-mxnet/tests
       linked by target "image-classification-predict" in directory /home/trz/github/incubator-mxnet/example/image-classification/predict-cpp
       linked by target "image-classification-predict" in directory /home/trz/github/incubator-mxnet/example/image-classification/predict-cpp
   
   -- Configuring incomplete, errors occurred!
   See also "/home/trz/github/incubator-mxnet/build/CMakeFiles/CMakeOutput.log".
   See also "/home/trz/github/incubator-mxnet/build/CMakeFiles/CMakeError.log".
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-568177120
 
 
   @leezu Thanks, I just did a straight checkout from github to get started and missed the recursive part from the docs.
   
   I still need a little more help with the install: 
   
   I upgraded cmake to 3.14 (3.10.2 with cuda 10 does not work, see: https://root-forum.cern.ch/t/intallation-cuda-cublas-device-library-advanced-set-to-notfound/33206/9 )
   
   I had to manually link liblapack.so too:
   
   sudo ln -s /usr/lib/x86_64-linux-gnu/liblapack.so.3 /usr/lib/liblapack.so
   
   to fix /usr/bin/ld: cannot find -llapack from ninja (also had to install ninja).
   
   I'm following this doc:
   
   https://mxnet.apache.org/get_started/build_from_source
   
   I've seen the ubuntu_core.sh and ubuntu_python.sh but I prefer to do it step by step and I'm also using mint 19.2.
   
   Now to add the module to my conda env I did:
   
   python python/setup.py install
   
   This added an mxnet to the pip list but not an mxnet-cu100.
   
   With only the mxnet module it does not work, if I ask for mxnet.__version__ I get:
   
   AttributeError: module 'mxnet' has no attribute '__version__'
   
   I manually added
   
   mxnet-cu100          1.6.0b20191102
   
   and it works (still with the broken OOM) but I think it is not the right thing to do.
   
   What are final steps? Is there a doc I missed?
   
   
   @ptrendx 
   Clone this project: 
   
   https://github.com/deepinsight/insightface
   
   Download and extract from here:
   
   https://www.dropbox.com/s/tj96fsm6t6rq8ye/model-r100-arcface-ms1m-refine-v2.zip?dl=0
   
   model-r100-ii into models keeping the subfolder.
   
   Copy the attached test into deploy and run it from that folder.
   
   [test_error.py.txt](https://github.com/apache/incubator-mxnet/files/3991369/test_error.py.txt)
   
   I've noticed that I get the error only if the batch size is greater than one.
   
   
   
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-568273050
 
 
   @leezu I added the mxnet-cuX after I saw this error:
   
   AttributeError: module 'mxnet' has no attribute 'cpu'
   
   Right after the build and install this is the situation:
   ```
   Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44) 
   [GCC 7.3.0] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import mxnet as mx
   >>> dir(mx)
   ['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
   >>> mx.__path__
   _NamespacePath(['/home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet-1.6.0-py3.6.egg/mxnet'])
   ```
   The module is found inside the conda env but is completely empty.
   
   The last line from the ninja build is this:
   ```
   750/750] : && /usr/bin/c++  -mf16c -Wall -Wno-unknown-pragmas -Wno-sign-compare -O3 -msse3 -std=c++11 -mf16c -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -fopenmp -std=c++0x -O3 -DNDEBUG  -rdynamic tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/engine_shutdown_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/omp_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/thread_local_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/threaded_engine_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/kvstore/gpu_topology_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/misc/base.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/misc/libinfo_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/activation_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/batchnorm_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/coreop_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/dropout_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/fully_conn_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/krprod_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/mkldnn_operator_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/mkldnn_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/runner/core_op_runner_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/slice_channel_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/tune/operator_tune_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/storage/storage_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/test_main.cc.o  -o tests/mxnet_unit_tests -L/usr/local/cuda/lib64 -Wl,-rpath,/usr/local/cuda/lib64:/home/trz/github/incubator-mxnet/build/3rdparty/openmp/runtime/src lib/libgtest.a -Wl,--whole-archive libmxnet.a -Wl,--no-whole-archive 3rdparty/dmlc-core/libdmlc.a 3rdparty/mkldnn/src/libdnnl.a /usr/local/cuda/lib64/libnvToolsExt.so -lmkl_rt /usr/local/cuda/lib64/libcudart.so /usr/local/cuda/lib64/libcurand.so /usr/local/cuda/lib64/libcublas.so /usr/local/cuda/lib64/libcudart.so /usr/local/cuda/lib64/libcurand.so /usr/local/cuda/lib64/libcublas.so /usr/local/cuda/lib64/libcudnn.so -lrt -ljemalloc /usr/lib/x86_64-linux-gnu/libopencv_highgui.so.3.2.0 3rdparty/openmp/runtime/src/libomp.so -lpthread -llapack -ljemalloc /usr/local/cuda/lib64/libcudnn.so -lcufft -lcusolver -lnvrtc -lcuda -lrt -lrt -lpthread -llapack -lcufft -lcusolver -lnvrtc -lcuda /usr/lib/x86_64-linux-gnu/libopencv_imgcodecs.so.3.2.0 /usr/lib/x86_64-linux-gnu/libopencv_imgproc.so.3.2.0 /usr/lib/x86_64-linux-gnu/libopencv_core.so.3.2.0 -ldl -lpthread && :
   ```
   Did the build complete correctly?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] ballcue commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
ballcue commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-579312728
 
 
   I seem to be having the same problem.
   ```
   File ..., line 225, in detect
       scores = net_out[idx].asnumpy()
     File "/opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py", line 1996, in asnumpy
       ctypes.c_size_t(data.size)))
     File "/opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/base.py", line 253, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [05:22:25] /home/travis/build/dmlc/mxnet-distro/mxnet-build/3rdparty/mshadow/mshadow/././././cuda/tensor_gpu-inl.cuh:110: Check failed: err == cudaSuccess (2 vs. 0) : Name: MapPlanKernel ErrStr:out of memory
   Stack trace:
     [bt] (0) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x4b04cb) [0x7f8f0701e4cb]
     [bt] (1) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x2f59431) [0x7f8f09ac7431]
     [bt] (2) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x31b61ee) [0x7f8f09d241ee]
     [bt] (3) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x31b9a16) [0x7f8f09d27a16]
     [bt] (4) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25db7a9) [0x7f8f091497a9]
     [bt] (5) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25e1a1a) [0x7f8f0914fa1a]
     [bt] (6) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25c1cd1) [0x7f8f0912fcd1] 
     [bt] (7) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25c51e0) [0x7f8f091331e0]
     [bt] (8) /opt/conda/envs/mlaas/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25c5476) [0x7f8f09133476]
   ```
   
   @lorenzob were you able to solve the out of memory errors? @szha , if the problem is in the call to mshadow, is there a way of working around mshadow to avoid it? Thanks.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-568273050
 
 
   @leezu I added the mxnet-cuX after I saw this error:
   
   AttributeError: module 'mxnet' has no attribute 'cpu'
   
   Right after the build and install this is the situation:
   
   Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44) 
   [GCC 7.3.0] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import mxnet as mx
   >>> dir(mx)
   ['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
   >>> mx.__path__
   _NamespacePath(['/home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet-1.6.0-py3.6.egg/mxnet'])
   
   The module is found inside pip but it is completely empty.
   
   The last line from the ninja build is this:
   ```
   750/750] : && /usr/bin/c++  -mf16c -Wall -Wno-unknown-pragmas -Wno-sign-compare -O3 -msse3 -std=c++11 -mf16c -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -fopenmp -std=c++0x -O3 -DNDEBUG  -rdynamic tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/engine_shutdown_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/omp_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/thread_local_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/threaded_engine_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/kvstore/gpu_topology_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/misc/base.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/misc/libinfo_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/activation_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/batchnorm_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/coreop_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/dropout_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/fully_conn_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/krprod_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/mkldnn_operator_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/mkldnn_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/runner/core_op_runner_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/slice_channel_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/tune/operator_tune_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/storage/storage_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/test_main.cc.o  -o tests/mxnet_unit_tests -L/usr/local/cuda/lib64 -Wl,-rpath,/usr/local/cuda/lib64:/home/trz/github/incubator-mxnet/build/3rdparty/openmp/runtime/src lib/libgtest.a -Wl,--whole-archive libmxnet.a -Wl,--no-whole-archive 3rdparty/dmlc-core/libdmlc.a 3rdparty/mkldnn/src/libdnnl.a /usr/local/cuda/lib64/libnvToolsExt.so -lmkl_rt /usr/local/cuda/lib64/libcudart.so /usr/local/cuda/lib64/libcurand.so /usr/local/cuda/lib64/libcublas.so /usr/local/cuda/lib64/libcudart.so /usr/local/cuda/lib64/libcurand.so /usr/local/cuda/lib64/libcublas.so /usr/local/cuda/lib64/libcudnn.so -lrt -ljemalloc /usr/lib/x86_64-linux-gnu/libopencv_highgui.so.3.2.0 3rdparty/openmp/runtime/src/libomp.so -lpthread -llapack -ljemalloc /usr/local/cuda/lib64/libcudnn.so -lcufft -lcusolver -lnvrtc -lcuda -lrt -lrt -lpthread -llapack -lcufft -lcusolver -lnvrtc -lcuda /usr/lib/x86_64-linux-gnu/libopencv_imgcodecs.so.3.2.0 /usr/lib/x86_64-linux-gnu/libopencv_imgproc.so.3.2.0 /usr/lib/x86_64-linux-gnu/libopencv_core.so.3.2.0 -ldl -lpthread && :
   ```
   Did the build complete correctly?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-579678023
 
 
   @ballcue No, but I did not do further tests with the latest versions since the last post.
   As an obvious workaround I reduced the batch size and, as my process is running under gunicorn, I kill the process in case of OOM so that a new one is respawned.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-568287777
 
 
   @leezu I did remove the all the mxnet packages and also created a new conda env to be sure.
   
   But I used the "setup.py install" script to install the module rather than pip directly. Now it works, thanks, but the error is still there:
   
   ```
   [...]
     File "/home/trz/progetti/zzzz/git_repo/ai-face-matching/arcface/helper.py", line 156, in detect_first_stage
       output = net.predict(input_buf)
     File "/home/trz/github/incubator-mxnet/python/mxnet/model.py", line 750, in predict
       o_list.append(o_nd[0:real_size].asnumpy())
     File "/home/trz/github/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 2552, in asnumpy
       ctypes.c_size_t(data.size)))
     File "/home/trz/github/incubator-mxnet/python/mxnet/base.py", line 278, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [18:48:23] /home/trz/github/incubator-mxnet/include/mshadow/././././cuda/tensor_gpu-inl.cuh:110: Check failed: err == cudaSuccess (2 vs. 0) : Name: MapPlanKernel ErrStr:out of memory
   
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-568273050
 
 
   @leezu I added the mxnet-cuX after I saw this error:
   
   AttributeError: module 'mxnet' has no attribute 'cpu'
   
   Right after the build and install this is the situation:
   ```
   Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44) 
   [GCC 7.3.0] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import mxnet as mx
   >>> dir(mx)
   ['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
   >>> mx.__path__
   _NamespacePath(['/home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet-1.6.0-py3.6.egg/mxnet'])
   ```
   The module is found inside the conda env but it is completely empty.
   
   The last line from the ninja build is this:
   ```
   750/750] : && /usr/bin/c++  -mf16c -Wall -Wno-unknown-pragmas -Wno-sign-compare -O3 -msse3 -std=c++11 -mf16c -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -fopenmp -std=c++0x -O3 -DNDEBUG  -rdynamic tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/engine_shutdown_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/omp_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/thread_local_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/threaded_engine_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/kvstore/gpu_topology_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/misc/base.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/misc/libinfo_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/activation_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/batchnorm_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/coreop_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/dropout_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/fully_conn_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/krprod_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/mkldnn_operator_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/mkldnn_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/runner/core_op_runner_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/slice_channel_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/tune/operator_tune_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/storage/storage_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/test_main.cc.o  -o tests/mxnet_unit_tests -L/usr/local/cuda/lib64 -Wl,-rpath,/usr/local/cuda/lib64:/home/trz/github/incubator-mxnet/build/3rdparty/openmp/runtime/src lib/libgtest.a -Wl,--whole-archive libmxnet.a -Wl,--no-whole-archive 3rdparty/dmlc-core/libdmlc.a 3rdparty/mkldnn/src/libdnnl.a /usr/local/cuda/lib64/libnvToolsExt.so -lmkl_rt /usr/local/cuda/lib64/libcudart.so /usr/local/cuda/lib64/libcurand.so /usr/local/cuda/lib64/libcublas.so /usr/local/cuda/lib64/libcudart.so /usr/local/cuda/lib64/libcurand.so /usr/local/cuda/lib64/libcublas.so /usr/local/cuda/lib64/libcudnn.so -lrt -ljemalloc /usr/lib/x86_64-linux-gnu/libopencv_highgui.so.3.2.0 3rdparty/openmp/runtime/src/libomp.so -lpthread -llapack -ljemalloc /usr/local/cuda/lib64/libcudnn.so -lcufft -lcusolver -lnvrtc -lcuda -lrt -lrt -lpthread -llapack -lcufft -lcusolver -lnvrtc -lcuda /usr/lib/x86_64-linux-gnu/libopencv_imgcodecs.so.3.2.0 /usr/lib/x86_64-linux-gnu/libopencv_imgproc.so.3.2.0 /usr/lib/x86_64-linux-gnu/libopencv_core.so.3.2.0 -ldl -lpthread && :
   ```
   Did the build complete correctly?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] leezu commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
leezu commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-568181729
 
 
   @lorenzob you need to uninstall all mxnet packages first (ie uninstall mxnet-cu100), and then install the source compiled version. It is expected that only `mxnet` package (and no mxnet-cuX) package is installed afterwards.
   
   The attribute error you experienced is due to having two versions installed (in my experience).
   
   Sorry to hear you experienced issues with the cmake & cuda setup. The requirements on a recent CMake version will be properly declared once https://github.com/apache/incubator-mxnet/pull/17031 is reviewed and merged.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] leezu commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
leezu commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-567937985
 
 
   @lorenzob thank you. Actually there is the `3rdparty/googletest` submodule included in MXNet. You need to run `git submodule update --init --recursive` or clone with `git clone --recursive https://github.com/apache/incubator-mxnet/` to make sure all submodules are initialized correctly.
   
   A better error message should be provided indeed..

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-567606406
 
 
   With the nightly first I got this error:
   
   https://github.com/apache/incubator-mxnet/issues/16785
   
   and I solved(?) it by setting MXNET_USE_FUSION=0.
   
   After that, on the first call after the OOM I get (from another model, not the same that threw the OOM error):
   ```
   Traceback (most recent call last):
     File "core/facerec.py", line 676, in <module>
       img_result, distances = compare(list(images_data), "work/")
     File "core/facerec.py", line 142, in compare
       images, aligned_images, detection_boxes = load_and_align_data(image_files, image_size, margin, gpu_memory_fraction, work_dir)
     File "core/facerec.py", line 368, in load_and_align_data
       result = run_detection(i, factor, image_size, img_data, margin, minsize, onet, pnet, rnet, threshold, options, work_dir)
     File "core/facerec.py", line 434, in run_detection
       ret = model.detector.detect_face(img, det_type=0)
     File "/home/trz/progetti/zzzz/git_repo/ai-face-matching/arcface/mtcnn_detector.py", line 365, in detect_face
       total_boxes.extend(local_boxes)
     File "/home/trz/progetti/zzzz/git_repo/ai-face-matching/arcface/helper.py", line 168, in detect_first_stage_warpper
       return detect_first_stage(*args)
     File "/home/trz/progetti/zzzz/git_repo/ai-face-matching/arcface/helper.py", line 156, in detect_first_stage
       output = net.predict(input_buf)
     File "/home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/model.py", line 736, in predict
       o_list.append(o_nd[0:real_size].asnumpy())
     File "/home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2532, in asnumpy
       ctypes.c_size_t(data.size)))
     File "/home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/base.py", line 255, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [19:21:31] /home/ubuntu/mxnet-distro/mxnet-build/3rdparty/mshadow/mshadow/././././cuda/tensor_gpu-inl.cuh:110: Check failed: err == cudaSuccess (2 vs. 0) : Name: MapPlanKernel ErrStr:out of memory
   Stack trace:
     [bt] (0) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x6a6deb) [0x7fbc5fe28deb]
     [bt] (1) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x6c91b27) [0x7fbc66413b27]
     [bt] (2) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x39edffe) [0x7fbc6316fffe]
     [bt] (3) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x37ca914) [0x7fbc62f4c914]
     [bt] (4) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x37d8941) [0x7fbc62f5a941]
     [bt] (5) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x37dbf60) [0x7fbc62f5df60]
     [bt] (6) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x37dc1f6) [0x7fbc62f5e1f6]
     [bt] (7) /home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x37d7094) [0x7fbc62f59094]
     [bt] (8) /home/trz/miniconda3/envs/facerec/bin/../lib/libstdc++.so.6(+0xb8678) [0x7fbcd2a55678]
   ```
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-569656522
 
 
   @leezu @ptrendx I used commit d000c3 that I think includes #17114
   
   I no longer get the "Operator is non-differentiable" error if I do not set the MXNET_USE_FUSION. Setting MXNET_USE_FUSION=0 never solved the OOM error.
   
   I still get the OOM if I use more than 10 112x112 images in one batch (for inference). When I get the first OOM I'm actually filling all the available free memory (about 3GB) and memory usage remains at near 100% after the first OOM exception (I added a time.sleep and checked this with nvidia-smi).
   
   The first OOM and the second one are different:
   
   First:
   ```
     File "core/facerec.py", line 184, in compare
       embedding = model.model.get_outputs()[0].asnumpy()
     File "/home/trz/github/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 2552, in asnumpy
       ctypes.c_size_t(data.size)))
     File "/home/trz/github/incubator-mxnet/python/mxnet/base.py", line 278, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [12:42:39] ../src/storage/./pooled_storage_manager.h:161: cudaMalloc retry failed: out of memory
   ```
   
   Second:
   ```
     File "/home/trz/progetti/zzzz/git_repo/ai-face-matching/arcface/helper.py", line 156, in detect_first_stage
       output = net.predict(input_buf)
     File "/home/trz/github/incubator-mxnet/python/mxnet/model.py", line 750, in predict
       o_list.append(o_nd[0:real_size].asnumpy())
     File "/home/trz/github/incubator-mxnet/python/mxnet/ndarray/ndarray.py", line 2552, in asnumpy
       ctypes.c_size_t(data.size)))
     File "/home/trz/github/incubator-mxnet/python/mxnet/base.py", line 278, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [12:39:32] /home/trz/github/incubator-mxnet/include/mshadow/././././cuda/tensor_gpu-inl.cuh:110: Check failed: err == cudaSuccess (2 vs. 0) : Name: MapPlanKernel ErrStr:out of memory
   ```
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] ptrendx commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
ptrendx commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-567985315
 
 
   @lorenzob Can you provide some repro instructions for the error where you needed to disable fusion?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] leezu commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
leezu commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-568728779
 
 
   Thanks for confirming @lorenzob. So you confirmed the issue disappears when setting `MXNET_USE_FUSION=0` with the code from #17114, correct?
   
   How close to out of memory are you with `MXNET_USE_FUSION=0`? (ie. what's the memory usage)
   
   CC @ptrendx

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] ballcue commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
ballcue commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-579899094
 
 
   @lorenzob Got it, thank you.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
lorenzob edited a comment on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-568273050
 
 
   @leezu I added the mxnet-cuX after I saw this error:
   
   AttributeError: module 'mxnet' has no attribute 'cpu'
   
   Right after the build and install this is the situation:
   ```
   Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44) 
   [GCC 7.3.0] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import mxnet as mx
   >>> dir(mx)
   ['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
   >>> mx.__path__
   _NamespacePath(['/home/trz/miniconda3/envs/facerec/lib/python3.6/site-packages/mxnet-1.6.0-py3.6.egg/mxnet'])
   ```
   The module is found inside pip but it is completely empty.
   
   The last line from the ninja build is this:
   ```
   750/750] : && /usr/bin/c++  -mf16c -Wall -Wno-unknown-pragmas -Wno-sign-compare -O3 -msse3 -std=c++11 -mf16c -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -fopenmp -std=c++0x -O3 -DNDEBUG  -rdynamic tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/engine_shutdown_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/omp_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/thread_local_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/engine/threaded_engine_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/kvstore/gpu_topology_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/misc/base.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/misc/libinfo_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/activation_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/batchnorm_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/coreop_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/dropout_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/fully_conn_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/krprod_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/mkldnn_operator_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/mkldnn_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/runner/core_op_runner_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/slice_channel_perf.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/operator/tune/operator_tune_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/storage/storage_test.cc.o tests/CMakeFiles/mxnet_unit_tests.dir/cpp/test_main.cc.o  -o tests/mxnet_unit_tests -L/usr/local/cuda/lib64 -Wl,-rpath,/usr/local/cuda/lib64:/home/trz/github/incubator-mxnet/build/3rdparty/openmp/runtime/src lib/libgtest.a -Wl,--whole-archive libmxnet.a -Wl,--no-whole-archive 3rdparty/dmlc-core/libdmlc.a 3rdparty/mkldnn/src/libdnnl.a /usr/local/cuda/lib64/libnvToolsExt.so -lmkl_rt /usr/local/cuda/lib64/libcudart.so /usr/local/cuda/lib64/libcurand.so /usr/local/cuda/lib64/libcublas.so /usr/local/cuda/lib64/libcudart.so /usr/local/cuda/lib64/libcurand.so /usr/local/cuda/lib64/libcublas.so /usr/local/cuda/lib64/libcudnn.so -lrt -ljemalloc /usr/lib/x86_64-linux-gnu/libopencv_highgui.so.3.2.0 3rdparty/openmp/runtime/src/libomp.so -lpthread -llapack -ljemalloc /usr/local/cuda/lib64/libcudnn.so -lcufft -lcusolver -lnvrtc -lcuda -lrt -lrt -lpthread -llapack -lcufft -lcusolver -lnvrtc -lcuda /usr/lib/x86_64-linux-gnu/libopencv_imgcodecs.so.3.2.0 /usr/lib/x86_64-linux-gnu/libopencv_imgproc.so.3.2.0 /usr/lib/x86_64-linux-gnu/libopencv_core.so.3.2.0 -ldl -lpthread && :
   ```
   Did the build complete correctly?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] leezu commented on issue #17126: Memory should be completely released after an OOM happens

Posted by GitBox <gi...@apache.org>.
leezu commented on issue #17126: Memory should be completely released after an OOM happens
URL: https://github.com/apache/incubator-mxnet/issues/17126#issuecomment-568278049
 
 
   @lorenzob I think the problem is that you didn't uninstall all mxnet versions obtained via pip, before installing the self-compiled version.
   
   Ie. first `pip uninstall mxnet`, `pip uninstall mxnet-cuX`, etc.
   Then `cd ~/path/to/mxnet-incubator/python; pip install --user -e .` to install the self-compiled version.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services