You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ulysses-you (via GitHub)" <gi...@apache.org> on 2024/05/24 09:51:36 UTC

[I] ArrowNativeMemoryPool memory leak [incubator-gluten]

ulysses-you opened a new issue, #5864:
URL: https://github.com/apache/incubator-gluten/issues/5864

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   From a [github action test ](https://github.com/apache/incubator-gluten/actions/runs/9220817939/job/25371831381?pr=5862)
   ```
   E20240524 09:48:03.909570  7753 Exceptions.h:67] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1858, Function:terminate, Expression:  Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
   24/05/24 09:48:03 WARN ArrowNativeMemoryPool: Arrow pool still reserved non-zero bytes, which may cause memory leak, size: 128.0 B. 
   /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/src/arrow_ep/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: Java Exception: java.lang.IncompatibleClassChangeError
   
   /tmp/jnilib-7232572254174836976.tmp(+0x12183e8)[0x7f5da665e3e8]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow4util8ArrowLogD1Ev+0xed)[0x7f5da665e56d]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow7dataset3jni31ReservationListenableMemoryPool4FreeEPhll+0x45d)[0x7f5da5c5409d]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10PoolBufferD0Ev+0x47)[0x7f5da68ea8f7]
   /tmp/jnilib-7232572254174836976.tmp(_ZNSt6vectorISt10shared_ptrIN5arrow6BufferEESaIS3_EED1Ev+0x6b)[0x7f5da5d681fb]
   /tmp/jnilib-7232572254174836976.tmp(_ZNSt23_Sp_counted_ptr_inplaceIN5arrow9ArrayDataESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0xda)[0x7f5da5e75e0a]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow17SimpleRecordBatchD1Ev+0x116)[0x7f5da67943d6]
   /tmp/jnilib-7232572254174836976.tmp(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x48)[0x7f5da5c4d758]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_11RecordBatchEEE20WrapResultOnComplete8CallbackINSC_14ThenOnCompleteIZNS_23DefaultIfEmptyGeneratorISB_EclEvEUt_NSC_17PassthruOnFailureISI_EEEEEEED0Ev+0x2e)[0x7f5da5c9e6ae]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x3a4)[0x7f5da6677124]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureISt10shared_ptrINS_11RecordBatchEEE12MarkFinishedENS_6ResultIS3_EE+0xda)[0x7f5da5cbec6a]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_11RecordBatchEEE20WrapResultOnComplete8CallbackINS_6detail16MarkNextFinishedISC_SC_Lb0ELb0EEEEEE6invokeES4_+0x56)[0x7f5da5cbf846]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl21RunOrScheduleCallbackERKSt10shared_ptrINS_10FutureImplEEONS2_14CallbackRecordEb+0x46)[0x7f5da66775f6]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10FutureImpl11AddCallbackENS_8internal6FnOnceIFvRKS0_EEENS_15CallbackOptionsE+0xcb)[0x7f5da6662b0b]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt8functionIFNS8_ISt10shared_ptrINS_11RecordBatchEEEEvEEE20WrapResultOnComplete8CallbackINSG_14ThenOnCompleteIZNS_20FutureFirstGeneratorISC_EclEvEUlRKSF_E_NSG_17PassthruOnFailureISO_EEEEEEE6invokeES4_+0xde)[0x7f5da5cbfd8e]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
   /tmp/jnilib-7232572254174836976.tmp(+0x83c23b)[0x7f5da5c8223b]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
   /tmp/jnilib-7232572254174836976.tmp(+0x82d04e)[0x7f5da5c7304e]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_3csv15StreamingReaderEEE20WrapResultOnComplete8CallbackINS_6detail16MarkNextFinishedISD_SD_Lb0ELb0EEEEEE6invokeES4_+0x164)[0x7f5da5c9fe34]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
   /tmp/jnilib-7232572254174836976.tmp(+0x119ba3b)[0x7f5da65e1a3b]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureINS_8internal5EmptyEE12MarkFinishedIS2_vEEvNS_6StatusE+0x94)[0x7f5da5cad6e4]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureINS0_5EmptyEE21WrapStatusyOnComplete8CallbackINS_6detail16MarkNextFinishedISA_SA_Lb1ELb1EEEEEE6invokeES4_+0x28)[0x7f5da5cad848]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl21RunOrScheduleCallbackERKSt10shared_ptrINS_10FutureImplEEONS2_14CallbackRecordEb+0x46)[0x7f5da66775f6]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10FutureImpl11AddCallbackENS_8internal6FnOnceIFvRKS0_EEENS_15CallbackOptionsE+0xcb)[0x7f5da6662b0b]
   /tmp/jnilib-7232572254174836976.tmp(_ZNK5arrow6FutureINS_8internal5EmptyEE11AddCallbackINS_6detail16MarkNextFinishedIS3_S3_Lb1ELb1EEENS3_21WrapStatusyOnComplete8CallbackIS7_EEEEvT_NS_15CallbackOptionsE+0x5b)[0x7f5da66253bb]
   /tmp/jnilib-7232572254174836976.tmp(+0x11c8543)[0x7f5da660e543]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureISt10shared_ptrINS_6BufferEEE12MarkFinishedENS_6ResultIS3_EE+0xda)[0x7f5da6029c4a]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_6BufferEEE20WrapResultOnComplete8CallbackINS_6detail16MarkNextFinishedISC_SC_Lb0ELb0EEEEEE6invokeES4_+0x56)[0x7f5da602e706]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl21RunOrScheduleCallbackERKSt10shared_ptrINS_10FutureImplEEONS2_14CallbackRecordEb+0x46)[0x7f5da66775f6]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10FutureImpl11AddCallbackENS_8internal6FnOnceIFvRKS0_EEENS_15CallbackOptionsE+0xcb)[0x7f5da6662b0b]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_6BufferEEE20WrapResultOnComplete8CallbackINSC_14ThenOnCompleteIZNS_21TransformingGeneratorISB_SB_E26TransformingGeneratorStateclEvEUlRKSB_E_NSC_17PassthruOnFailureISL_EEEEEEE6invokeES4_+0x99)[0x7f5da6623e29]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureISt10shared_ptrINS_6BufferEEE12MarkFinishedENS_6ResultIS3_EE+0xda)[0x7f5da6029c4a]
   /tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvvEE6FnImplIZZNS0_8Executor10DoTransferISt10shared_ptrINS_6BufferEENS_6FutureIS9_EENS_6ResultIS9_EEEENSA_IT_EESF_bENUlRKSD_E0_clESH_EUlvE_E6invokeEv+0x52)[0x7f5da602e5e2]
   /tmp/jnilib-7232572254174836976.tmp(+0x1210abd)[0x7f5da6656abd]
   ```
   
   ### Spark version
   
   None
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] ArrowNativeMemoryPool memory leak [incubator-gluten]

Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #5864:
URL: https://github.com/apache/incubator-gluten/issues/5864#issuecomment-2130379056

   > cc @zhztheplayer @PHILO-HE is this a known issue ?
   
   I don't think so


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] ArrowNativeMemoryPool memory leak [incubator-gluten]

Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on issue #5864:
URL: https://github.com/apache/incubator-gluten/issues/5864#issuecomment-2129121349

   cc @zhztheplayer @PHILO-HE is this a known issue ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org