You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/02/21 22:40:17 UTC
[I] Columnar shuffle uses wrong memory allocator in unified memory mode [datafusion-comet]
andygrove opened a new issue, #1438:
URL: https://github.com/apache/datafusion-comet/issues/1438
### Describe the bug
I am using unified memory management:
```
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g \
```
I deliberately allocated a small amount of memory for testing purposes. However, the stack trace below shows that Comet is using `CometTestShuffleMemoryAllocator` which should not be the case.
```
Caused by: org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire 16777216 bytes of memory, got 2588672 bytes. Available: 2588672
at org.apache.spark.shuffle.comet.CometTestShuffleMemoryAllocator.allocateMemoryBlock(CometTestShuffleMemoryAllocator.java:129)
at org.apache.spark.shuffle.comet.CometTestShuffleMemoryAllocator.allocate(CometTestShuffleMemoryAllocator.java:116)
at org.apache.spark.sql.comet.execution.shuffle.SpillWriter.initialCurrentPage(SpillWriter.java:158)
at org.apache.spark.shuffle.sort.CometShuffleExternalSorter.insertRecord(CometShuffleExternalSorter.java:374)
at org.apache.spark.sql.comet.execution.shuffle.CometUnsafeShuffleWriter.insertRecordIntoSorter(CometUnsafeShuffleWriter.java:278)
at org.apache.spark.sql.comet.execution.shuffle.CometUnsafeShuffleWriter.write(CometUnsafeShuffleWriter.java:206)
```
This is created in the following code in `CometShuffleMemoryAllocatorTrait`:
```scala
boolean useUnifiedMemAllocator =
(boolean)
CometConf$.MODULE$.COMET_COLUMNAR_SHUFFLE_UNIFIED_MEMORY_ALLOCATOR_IN_TEST().get();
if (!useUnifiedMemAllocator) {
synchronized (CometShuffleMemoryAllocator.class) {
if (INSTANCE == null) {
// CometTestShuffleMemoryAllocator handles pages by itself so it can be a singleton.
INSTANCE = new CometTestShuffleMemoryAllocator(conf, taskMemoryManager, pageSize);
}
}
return INSTANCE;
```
It seems that the wrong config is being checked here.
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Columnar shuffle uses wrong memory allocator in unified memory mode [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #1438:
URL: https://github.com/apache/datafusion-comet/issues/1438#issuecomment-2675884654
This is a regression introduced in https://github.com/apache/datafusion-comet/commit/e72beb1faaba39d45e05e537d9b84151db7e73ff#diff-dcc1523ab10f465921fcdea05347484ed7bb944a546a0bb4902001f50da92b92L55
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Columnar shuffle uses wrong memory allocator in unified memory mode [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #1438: Columnar shuffle uses wrong memory allocator in unified memory mode
URL: https://github.com/apache/datafusion-comet/issues/1438
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Columnar shuffle uses wrong memory allocator in unified memory mode [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on issue #1438:
URL: https://github.com/apache/datafusion-comet/issues/1438#issuecomment-2695089326
I will take a look at this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org