You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ulysses-you (via GitHub)" <gi...@apache.org> on 2024/05/24 09:51:36 UTC
[I] ArrowNativeMemoryPool memory leak [incubator-gluten]
ulysses-you opened a new issue, #5864:
URL: https://github.com/apache/incubator-gluten/issues/5864
### Backend
VL (Velox)
### Bug description
From a [github action test ](https://github.com/apache/incubator-gluten/actions/runs/9220817939/job/25371831381?pr=5862)
```
E20240524 09:48:03.909570 7753 Exceptions.h:67] Line: /__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/velox/exec/Task.cpp:1858, Function:terminate, Expression: Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
24/05/24 09:48:03 WARN ArrowNativeMemoryPool: Arrow pool still reserved non-zero bytes, which may cause memory leak, size: 128.0 B.
/__w/incubator-gluten/incubator-gluten/ep/build-velox/build/velox_ep/_build/release/third_party/arrow_ep/src/arrow_ep/java/dataset/src/main/cpp/jni_util.cc:79: Failed to update reservation while freeing bytes: Java Exception: java.lang.IncompatibleClassChangeError
/tmp/jnilib-7232572254174836976.tmp(+0x12183e8)[0x7f5da665e3e8]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow4util8ArrowLogD1Ev+0xed)[0x7f5da665e56d]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow7dataset3jni31ReservationListenableMemoryPool4FreeEPhll+0x45d)[0x7f5da5c5409d]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10PoolBufferD0Ev+0x47)[0x7f5da68ea8f7]
/tmp/jnilib-7232572254174836976.tmp(_ZNSt6vectorISt10shared_ptrIN5arrow6BufferEESaIS3_EED1Ev+0x6b)[0x7f5da5d681fb]
/tmp/jnilib-7232572254174836976.tmp(_ZNSt23_Sp_counted_ptr_inplaceIN5arrow9ArrayDataESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv+0xda)[0x7f5da5e75e0a]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow17SimpleRecordBatchD1Ev+0x116)[0x7f5da67943d6]
/tmp/jnilib-7232572254174836976.tmp(_ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE10_M_releaseEv+0x48)[0x7f5da5c4d758]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_11RecordBatchEEE20WrapResultOnComplete8CallbackINSC_14ThenOnCompleteIZNS_23DefaultIfEmptyGeneratorISB_EclEvEUt_NSC_17PassthruOnFailureISI_EEEEEEED0Ev+0x2e)[0x7f5da5c9e6ae]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x3a4)[0x7f5da6677124]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureISt10shared_ptrINS_11RecordBatchEEE12MarkFinishedENS_6ResultIS3_EE+0xda)[0x7f5da5cbec6a]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_11RecordBatchEEE20WrapResultOnComplete8CallbackINS_6detail16MarkNextFinishedISC_SC_Lb0ELb0EEEEEE6invokeES4_+0x56)[0x7f5da5cbf846]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl21RunOrScheduleCallbackERKSt10shared_ptrINS_10FutureImplEEONS2_14CallbackRecordEb+0x46)[0x7f5da66775f6]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10FutureImpl11AddCallbackENS_8internal6FnOnceIFvRKS0_EEENS_15CallbackOptionsE+0xcb)[0x7f5da6662b0b]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt8functionIFNS8_ISt10shared_ptrINS_11RecordBatchEEEEvEEE20WrapResultOnComplete8CallbackINSG_14ThenOnCompleteIZNS_20FutureFirstGeneratorISC_EclEvEUlRKSF_E_NSG_17PassthruOnFailureISO_EEEEEEE6invokeES4_+0xde)[0x7f5da5cbfd8e]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(+0x83c23b)[0x7f5da5c8223b]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(+0x82d04e)[0x7f5da5c7304e]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_3csv15StreamingReaderEEE20WrapResultOnComplete8CallbackINS_6detail16MarkNextFinishedISD_SD_Lb0ELb0EEEEEE6invokeES4_+0x164)[0x7f5da5c9fe34]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(+0x119ba3b)[0x7f5da65e1a3b]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureINS_8internal5EmptyEE12MarkFinishedIS2_vEEvNS_6StatusE+0x94)[0x7f5da5cad6e4]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureINS0_5EmptyEE21WrapStatusyOnComplete8CallbackINS_6detail16MarkNextFinishedISA_SA_Lb1ELb1EEEEEE6invokeES4_+0x28)[0x7f5da5cad848]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl21RunOrScheduleCallbackERKSt10shared_ptrINS_10FutureImplEEONS2_14CallbackRecordEb+0x46)[0x7f5da66775f6]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10FutureImpl11AddCallbackENS_8internal6FnOnceIFvRKS0_EEENS_15CallbackOptionsE+0xcb)[0x7f5da6662b0b]
/tmp/jnilib-7232572254174836976.tmp(_ZNK5arrow6FutureINS_8internal5EmptyEE11AddCallbackINS_6detail16MarkNextFinishedIS3_S3_Lb1ELb1EEENS3_21WrapStatusyOnComplete8CallbackIS7_EEEEvT_NS_15CallbackOptionsE+0x5b)[0x7f5da66253bb]
/tmp/jnilib-7232572254174836976.tmp(+0x11c8543)[0x7f5da660e543]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureISt10shared_ptrINS_6BufferEEE12MarkFinishedENS_6ResultIS3_EE+0xda)[0x7f5da6029c4a]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_6BufferEEE20WrapResultOnComplete8CallbackINS_6detail16MarkNextFinishedISC_SC_Lb0ELb0EEEEEE6invokeES4_+0x56)[0x7f5da602e706]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl21RunOrScheduleCallbackERKSt10shared_ptrINS_10FutureImplEEONS2_14CallbackRecordEb+0x46)[0x7f5da66775f6]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow10FutureImpl11AddCallbackENS_8internal6FnOnceIFvRKS0_EEENS_15CallbackOptionsE+0xcb)[0x7f5da6662b0b]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvRKNS_10FutureImplEEE6FnImplINS_6FutureISt10shared_ptrINS_6BufferEEE20WrapResultOnComplete8CallbackINSC_14ThenOnCompleteIZNS_21TransformingGeneratorISB_SB_E26TransformingGeneratorStateclEvEUlRKSB_E_NSC_17PassthruOnFailureISL_EEEEEEE6invokeES4_+0x99)[0x7f5da6623e29]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow18ConcreteFutureImpl22DoMarkFinishedOrFailedENS_11FutureStateE+0x397)[0x7f5da6677117]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow6FutureISt10shared_ptrINS_6BufferEEE12MarkFinishedENS_6ResultIS3_EE+0xda)[0x7f5da6029c4a]
/tmp/jnilib-7232572254174836976.tmp(_ZN5arrow8internal6FnOnceIFvvEE6FnImplIZZNS0_8Executor10DoTransferISt10shared_ptrINS_6BufferEENS_6FutureIS9_EENS_6ResultIS9_EEEENSA_IT_EESF_bENUlRKSD_E0_clESH_EUlvE_E6invokeEv+0x52)[0x7f5da602e5e2]
/tmp/jnilib-7232572254174836976.tmp(+0x1210abd)[0x7f5da6656abd]
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] ArrowNativeMemoryPool memory leak [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #5864:
URL: https://github.com/apache/incubator-gluten/issues/5864#issuecomment-2130379056
> cc @zhztheplayer @PHILO-HE is this a known issue ?
I don't think so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] ArrowNativeMemoryPool memory leak [incubator-gluten]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on issue #5864:
URL: https://github.com/apache/incubator-gluten/issues/5864#issuecomment-2129121349
cc @zhztheplayer @PHILO-HE is this a known issue ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org