You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2024/12/27 06:43:28 UTC
[I] Memory leak issue [incubator-gluten]
wangyum opened a new issue, #8362:
URL: https://github.com/apache/incubator-gluten/issues/8362
### Backend
VL (Velox)
### Bug description
Log:
```
E20241226 23:24:39.978191 324668 VeloxMemoryManager.cc:401] Failed to release Velox memory manager after 43350ms as there are still outstanding memory resources.
E20241226 23:24:39.978293 324668 MemoryPool.cpp:442] [MEM] Memory leak (Used memory): Memory Pool[default_leaf LEAF root[root] parent[root] MALLOC track-usage thread-safe]<unlimited max capacity capacity 1.51GB used 1.50GB available 7.99MB reservation [used 1.50GB, reserved 1.51GB, min 0B] counters [allocs 2, frees 0, reserves 0, releases 0, collisions 0])>
E20241226 23:24:39.979336 324668 Exceptions.h:66] Line: /work/cpp/velox/memory/VeloxMemoryManager.cc:102, Function:removePool, Expression: pool->reservedBytes() == 0 (1619001344 vs. 0), Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (1619001344 vs. 0)
Retriable: False
Expression: pool->reservedBytes() == 0
Function: removePool
File: /work/cpp/velox/memory/VeloxMemoryManager.cc
Line: 102
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN6gluten20ListenableArbitrator10removePoolEPN8facebook5velox6memory10MemoryPoolE
# 4 _ZN8facebook5velox6memory13MemoryManager8dropPoolEPNS1_10MemoryPoolE
# 5 _ZN8facebook5velox6memory14MemoryPoolImplD2Ev
# 6 _ZN6gluten18VeloxMemoryManagerD1Ev
# 7 _ZN6gluten18VeloxMemoryManagerD0Ev
# 8 _ZN6gluten13MemoryManager7releaseEPS0_
# 9 Java_org_apache_gluten_memory_NativeMemoryManagerJniWrapper_release
# 10 0x00007ff6b0bc7be4
.
Driver stacktrace:
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:45)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:341)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:198)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:83)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:55)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:198)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:193)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:207)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 152 in stage 9.0 failed 4 times, most recent failure: Lost task 152.3 in stage 9.0 (TID 24199, 10.10.1.127, executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: Container from a bad node: container_e2320_1734105600630_7003_01_000015 on host: 10.10.1.127. Exit status: 134. Diagnostics: :61)
at org.apache.gluten.shaded.org.apache.arrow.memory.BaseAllocator.wrapForeignAllocation(BaseAllocator.java:241)
at org.apache.gluten.shaded.org.apache.arrow.memory.RootAllocator.wrapForeignAllocation(RootAllocator.java:29)
at org.apache.arrow.c.ReferenceCountedArrowArray.unsafeAssociateAllocation(ReferenceCountedArrowArray.java:67)
at org.apache.arrow.c.BufferImportTypeVisitor.importBuffer(BufferImportTypeVisitor.java:99)
at org.apache.arrow.c.BufferImportTypeVisitor.importData(BufferImportTypeVisitor.java:120)
at org.apache.arrow.c.BufferImportTypeVisitor.visit(BufferImportTypeVisitor.java:208)
at org.apache.arrow.c.BufferImportTypeVisitor.visit(BufferImportTypeVisitor.java:61)
at org.apache.gluten.shaded.org.apache.arrow.vector.types.pojo.ArrowType$Utf8.accept(ArrowType.java:739)
at org.apache.arrow.c.ArrayImporter.doImport(ArrayImporter.java:126)
... 138 more
```
### Spark version
Spark-3.5.x
### Spark configurations
key|value
-- | --
spark.gluten.memory.conservative.task.offHeap.size.in.bytes | 1640438897
spark.executor.memory|45g
spark.executor.cores|18
spark.gluten.memory.dynamic.offHeap.sizing.enabled | false
spark.gluten.memory.offHeap.size.in.bytes | 59055800320
spark.gluten.memory.task.offHeap.size.in.bytes | 3280877795
spark.gluten.memoryOverhead.size.in.bytes | 15032385536
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Memory leak issue [incubator-gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #8362:
URL: https://github.com/apache/incubator-gluten/issues/8362#issuecomment-2563666469
Thanks for reporting. Which version of Gluten is used here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] Memory leak issue [incubator-gluten]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on issue #8362:
URL: https://github.com/apache/incubator-gluten/issues/8362#issuecomment-2564181099
@zhztheplayer Master branch: https://github.com/wangyum/incubator-gluten/tree/ebay-build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org