You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wForget (via GitHub)" <gi...@apache.org> on 2025/05/23 10:07:53 UTC
[I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
wForget opened a new issue, #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737
### Backend
VL (Velox)
### Bug description
error msg:
```
ExecutorLostFailure (executor 38 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding physical memory limits. 4.1 GB of 4 GB physical memory used. Consider boosting spark.executor.memoryOverhead.
Driver stacktrace:
```
jemalloc profile:
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2920784548
We need to track the memory in offheap. @JkSelf Is there any other place in parquet writer not tracked? Like the compression memory.
@duanmeng's solution looks too complex. Can we pass a wrapped memory pool directly? We should already have the arrow wrapper. @zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2921266168
@FelixYBW Yes. The Parquet table writer currently passes the Arrow memory pool into Arrow::Writer, resulting in this memory not being tracked by the Velox memory manager. Since we have integrated the Arrow writer code into the Velox codebase, we can directly modify the memory allocation code path to utilize Velox buffers, following @duanmeng's approach.
@zhztheplayer , do you have any suggestions on how we can effectively replace the Arrow memory pool with Velox memory management? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2903950008
Memory used by `arrow::ImportRecordBatch` does not seem to be reflected in `TaskMemoryManager`:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2910918574
> [@wForget](https://github.com/wForget) Yes, these memory allocations are not tracked by the Velox memory manager, which has resulted in killed by yarn exception.
Thank you for your confirmation.
> Could we consider adjusting the block size and block rows parameters to reduce the memory footprint [#8816](https://github.com/apache/incubator-gluten/pull/8816)?
Adjusting the block size/rows should be able to avoid it. We only have a few jobs that have this problem, and I'd like to find a more general solution.
> I will review this issue [facebookincubator/velox#6985](https://github.com/facebookincubator/velox/issues/6985) to evaluate the effort involved. Thanks.
Thanks, that looks like a good solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2921503436
We could consider using Gluten's [ArrowMemoryPool.h](https://github.com/apache/incubator-gluten/blob/main/cpp/core/memory/ArrowMemoryPool.h) to encapsulate Velox's MemoryAllocator for memory allocation, replacing Arrow's MemoryPool. This approach would allow Velox's memory manager to track the allocated memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2910869990
@wForget Yes, these memory allocations are not tracked by the Velox memory manager, which has resulted in killed by yarn exception. Could we consider adjusting the block size and block rows parameters to reduce the memory footprint https://github.com/apache/incubator-gluten/pull/8816? I will review this issue https://github.com/facebookincubator/velox/issues/6985 to evaluate the effort involved. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2923874141
@JkSelf we should be be able to pass the ArrowMemoryPool to parquet write directly. Not touching the velox buffer things.
`ArrowMemoryPool::Allocate -> ListenableMemoryAllocator::allocate -> SparkAllocationListener::allocationChanged -> spark-executor-memory-track`
Our shuffle uses this memory pool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2933212782
> [@FelixYBW](https://github.com/FelixYBW) Yes, I understand your approach. I will look into how to pass ArrowMemoryPool into the Parquet writing process later. Thanks.
We need the peak value, let's see if we should merge the shufflememorypool and arrowmemorypool
#9836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2953161920
#9869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2908610795
Possibly related issue: https://github.com/facebookincubator/velox/issues/6985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf closed issue #9737: [VL][1.2] OOM kill by yarn during parquet write
URL: https://github.com/apache/incubator-gluten/issues/9737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2904002201
Do we need to override `ARROW_DEFAULT_MEMORY_POOL`? https://arrow.apache.org/docs/cpp/memory.html#overriding-the-default-memory-pool
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2908827527
Cc: @JkSelf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL][1.2] OOM kill by yarn during parquet write [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #9737:
URL: https://github.com/apache/incubator-gluten/issues/9737#issuecomment-2933167531
@FelixYBW Yes, I understand your approach. I will look into how to pass ArrowMemoryPool into the Parquet writing process later. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org