You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/01/14 05:23:22 UTC
[GitHub] xuw080 opened a new issue #9416: Out of memory
xuw080 opened a new issue #9416: Out of memory
URL: https://github.com/apache/incubator-mxnet/issues/9416
## Error Message:
(Paste the complete error message, including stack trace.)
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data bbox_target_stride32 has a shape [1,576], which is larger than already allocated shape [1,108]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data bbox_target_stride16 has a shape [1,972], which is larger than already allocated shape [1,864]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data bbox_target_stride4 has a shape [1,9000], which is larger than already allocated shape [1,8604]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data mask_target_stride32 has a shape [1,16,9,28,28], which is larger than already allocated shape [1,3,9,28,28]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data mask_target_stride16 has a shape [1,27,9,28,28], which is larger than already allocated shape [1,24,9,28,28]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data mask_target_stride4 has a shape [1,250,9,28,28], which is larger than already allocated shape [1,239,9,28,28]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data mask_weight_stride32 has a shape [1,16,9,1,1], which is larger than already allocated shape [1,3,9,1,1]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data mask_weight_stride16 has a shape [1,27,9,1,1], which is larger than already allocated shape [1,24,9,1,1]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data mask_weight_stride4 has a shape [1,250,9,1,1], which is larger than already allocated shape [1,239,9,1,1]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] src/executor/graph_executor.cc:706: Bucketing: data d_label has a shape [1,319,1], which is larger than already allocated shape [1,318,1]. Need to re-allocate. Consider putting default bucket key to be the bucket taking the largest input for better memory sharing.
[21:14:30] /data4/xuw080/mx-maskrcnn/incubator-mxnet/dmlc-core/include/dmlc/logging.h:308: [21:14:30] src/storage/./pooled_storage_manager.h:102: cudaMalloc failed: out of memory
Stack trace returned 10 entries:
[bt] (0) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f0ad80b20bc]
[bt] (1) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7storage23GPUPooledStorageManager5AllocEm+0x1d8) [0x7f0ada6bbc98]
[bt] (2) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet11StorageImpl5AllocEmNS_7ContextE+0x61) [0x7f0ada6bf0b1]
[bt] (3) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZNK5mxnet7NDArray13CheckAndAllocEv+0x34a) [0x7f0ad835363a]
[bt] (4) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec23StatefulComputeExecutor3RunENS_10RunContextEb+0xbd0) [0x7f0ada18bd00]
[bt] (5) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(+0x29748e7) [0x7f0ada1908e7]
[bt] (6) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x9d) [0x7f0ada17337d]
[bt] (7) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine23ThreadedEnginePerDevice9GPUWorkerILN4dmlc19ConcurrentQueueTypeE0EEEvNS_7ContextEbPNS1_17ThreadWorkerBlockIXT_EEESt10shared_ptrINS0_10ThreadPool11SimpleEventEE+0xf3) [0x7f0ada177503]
[bt] (8) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_+0x56) [0x7f0ada1776e6]
[bt] (9) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x3b) [0x7f0ada17491b]
[21:14:30] /data4/xuw080/mx-maskrcnn/incubator-mxnet/dmlc-core/include/dmlc/logging.h:308: [21:14:30] src/engine/./threaded_engine.h:370: [21:14:30] src/storage/./pooled_storage_manager.h:102: cudaMalloc failed: out of memory
Stack trace returned 10 entries:
[bt] (0) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f0ad80b20bc]
[bt] (1) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7storage23GPUPooledStorageManager5AllocEm+0x1d8) [0x7f0ada6bbc98]
[bt] (2) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet11StorageImpl5AllocEmNS_7ContextE+0x61) [0x7f0ada6bf0b1]
[bt] (3) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZNK5mxnet7NDArray13CheckAndAllocEv+0x34a) [0x7f0ad835363a]
[bt] (4) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec23StatefulComputeExecutor3RunENS_10RunContextEb+0xbd0) [0x7f0ada18bd00]
[bt] (5) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(+0x29748e7) [0x7f0ada1908e7]
[bt] (6) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x9d) [0x7f0ada17337d]
[bt] (7) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine23ThreadedEnginePerDevice9GPUWorkerILN4dmlc19ConcurrentQueueTypeE0EEEvNS_7ContextEbPNS1_17ThreadWorkerBlockIXT_EEESt10shared_ptrINS0_10ThreadPool11SimpleEventEE+0xf3) [0x7f0ada177503]
[bt] (8) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_+0x56) [0x7f0ada1776e6]
[bt] (9) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x3b) [0x7f0ada17491b]
A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 8 entries:
[bt] (0) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f0ad80b20bc]
[bt] (1) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x3a0) [0x7f0ada173680]
[bt] (2) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet6engine23ThreadedEnginePerDevice9GPUWorkerILN4dmlc19ConcurrentQueueTypeE0EEEvNS_7ContextEbPNS1_17ThreadWorkerBlockIXT_EEESt10shared_ptrINS0_10ThreadPool11SimpleEventEE+0xf3) [0x7f0ada177503]
[bt] (3) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_+0x56) [0x7f0ada1776e6]
[bt] (4) /data4/xuw080/mx-maskrcnn/incubator-mxnet/python/mxnet/../../lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x3b) [0x7f0ada17491b]
[bt] (5) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb1a60) [0x7f0abd5f0a60]
[bt] (6) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f0ae449a184]
[bt] (7) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f0ae41c6ffd]
Tried many methods, but can not figure out what is the meaning of this error?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services