You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "liujiayi771 (via GitHub)" <gi...@apache.org> on 2025/12/29 00:45:45 UTC
[I] Memory leak for ColumnarPartialProject [incubator-gluten]
liujiayi771 opened a new issue, #11336:
URL: https://github.com/apache/incubator-gluten/issues/11336
### Backend
VL (Velox)
### Bug description
If spark.gluten.sql.columnar.partial.project = true
```shell
velox_ep/velox/exec/Task.cpp:2409, Function:terminate, Expression: Cancelled, Source: RUNTIME, ErrorCode: INVALID_STATE
E20251229 02:43:05.733094 177 MemoryPool.cpp:460] [MEM] Memory leak (Used memory): Memory Pool[default_leaf LEAF root[root] parent[root] MALLOC track-usage thread-safe]<unlimited max capacity capacity 1.19GB used 1.18GB available 3.60MB reservation [used 1.18GB, reserved 1.19GB, min 0B] counters [allocs 4230, frees 4140, reserves 0, releases 0, collisions 0])>
E20251229 02:43:05.733157 177 Exceptions.h:66] Line: gluten/cpp/velox/memory/VeloxMemoryManager.cc:110, Function:removePool, Expression: pool->reservedBytes() == 0 (1275068416 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (1275068416 vs. 0)
Retriable: False
Expression: pool->reservedBytes() == 0
Function: removePool
File: gluten/cpp/velox/memory/VeloxMemoryManager.cc
Line: 110
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN6gluten12_GLOBAL__N_120ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE
# 4 _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE
# 5 _ZN8facebook5velox6memory14MemoryPoolImplD2Ev
# 6 _ZNSt16_Sp_counted_baseILN9__gnu_cxx12_Lock_policyE2EE24_M_release_last_use_coldEv
# 7 _ZN6gluten18VeloxMemoryManagerD1Ev
# 8 _ZN6gluten18VeloxMemoryManagerD0Ev
# 9 _ZN6gluten13MemoryManager7releaseEPS0_
# 10 Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
# 11 0x00007f983a1e445c
```
After disabling spark.gluten.sql.columnar.partial.project, the issue disappeared.
### Gluten version
Gluten-1.5
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Memory leak for ColumnarPartialProject [incubator-gluten]
Posted by "liujiayi771 (via GitHub)" <gi...@apache.org>.
liujiayi771 commented on issue #11336:
URL: https://github.com/apache/incubator-gluten/issues/11336#issuecomment-3717074598
@wForget Yes. The code base is 2025.08
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Memory leak for ColumnarPartialProject [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #11336:
URL: https://github.com/apache/incubator-gluten/issues/11336#issuecomment-3713237071
Do you include this fix https://github.com/apache/incubator-gluten/pull/10088?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org