You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "ccat3z (via GitHub)" <gi...@apache.org> on 2024/11/22 08:17:16 UTC
[I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
ccat3z opened a new issue, #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025
### Backend
VL (Velox)
### Bug description
Distinct aggregation will merge all sorted spill file in `getOutput()` (`SpillPartition::createOrderedReader`). If there are too many spill files, reading the first batch of each file into memory will consume a significant amount of memory. In one of our internal cases, one task generated 300 spill files, which requires close to 3G of memory.
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "ccat3z (via GitHub)" <gi...@apache.org>.
ccat3z commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2493146572
A possible workaround is to reduce `kSpillWriteBufferSize`. Why it is set to 4M by default? Is there any experience in performance tuning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499406108
> > In that case, we need to respect UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE. Velox spill RowVector, so we must read the `buffer size` ahead. But it's better to add a config for total buffer size when UNSAFE_SORTER_SPILL_READ_AHEAD_ENABLED is false.
>
> So the worst case of Vanilla spark is also 1M buffer per file, right? Let's hornor the value of spark.unsafe.sorter.spill.reader.buffer.size then. It may be set in queries.
Yes, I will draft a PR to respect this config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499333602
> The kSpillReadBufferSize controls the size to read each file, the ordered reader will create FileOutputStream for each file and allocate kSpillReadBufferSize (default 1MB) for one file. Can you try to adjust this value? Maybe we should add a new config to control the read buffer size for all the files. numFiles * bufferSize < threshold.
should we propose the way of https://github.com/apache/incubator-gluten/pull/7861?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499255836
Thank you, @jinchengchenghh . With the tuning of kMaxSpillRunRows and kSpillWriteBufferSize. one of my task succeed but the other one still fails. Looks like it still have some large memory allocation in getoutput.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499512268
Spark closes the reader when `loadNext` after all the records consumed.
I would prefer to close the input file after file is `atEnd`, https://github.com/facebookincubator/velox/blob/main/velox/exec/SpillFile.cpp#L314
It may benefits release the memory as early as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2495385404
Thank you, @ccat3z . I encounted the same issue in order by operator and debugged several days!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2496534497
Spark also open all the spill file to read.
```
final UnsafeSorterSpillMerger spillMerger = new UnsafeSorterSpillMerger(
recordComparatorSupplier.get(), prefixComparator, spillWriters.size());
for (UnsafeSorterSpillWriter spillWriter : spillWriters) {
spillMerger.addSpillIfNotEmpty(spillWriter.getReader(serializerManager));
}
if (inMemSorter != null) {
readingIterator = new SpillableIterator(inMemSorter.getSortedIterator());
spillMerger.addSpillIfNotEmpty(readingIterator);
}
return spillMerger.getSortedIterator();
```
Spark use the `PriorityQueue<UnsafeSorterIterator> priorityQueue` to get the record to merge.
```
Comparator<UnsafeSorterIterator> comparator = (left, right) -> {
int prefixComparisonResult =
prefixComparator.compare(left.getKeyPrefix(), right.getKeyPrefix());
if (prefixComparisonResult == 0) {
return recordComparator.compare(
left.getBaseObject(), left.getBaseOffset(), left.getRecordLength(),
right.getBaseObject(), right.getBaseOffset(), right.getRecordLength());
} else {
return prefixComparisonResult;
}
};
priorityQueue = new PriorityQueue<>(numSpills, comparator);
```
It has config to control the read buffer size (default 1 MB) as following:
```
private[spark] val UNSAFE_SORTER_SPILL_READ_AHEAD_ENABLED =
ConfigBuilder("spark.unsafe.sorter.spill.read.ahead.enabled")
.internal()
.version("2.3.0")
.booleanConf
.createWithDefault(true)
private[spark] val UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE =
ConfigBuilder("spark.unsafe.sorter.spill.reader.buffer.size")
.internal()
.version("2.1.0")
.bytesConf(ByteUnit.BYTE)
.checkValue(v => 1024 * 1024 <= v && v <= MAX_BUFFER_SIZE_BYTES,
s"The value must be in allowed range [1,048,576, ${MAX_BUFFER_SIZE_BYTES}].")
.createWithDefault(1024 * 1024)
```
class `UnsafeSorterSpillReader`
```
if (readAheadEnabled) {
this.in = new ReadAheadInputStream(serializerManager.wrapStream(blockId, bs),
bufferSizeBytes);
} else {
this.in = serializerManager.wrapStream(blockId, bs);
}
this.din = new DataInputStream(this.in);
```
It only needs to load one record at one time, after loaded, it will put the `UnsafeSorterIterator reader` to the priorityQueue again to load next record.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499299722
> The kSpillReadBufferSize controls the size to read each file, the ordered reader will create FileOutputStream for each file and allocate kSpillReadBufferSize (default 1MB) for one file. Can you try to adjust this value? Maybe we should add a new config to control the read buffer size for all the files. numFiles * bufferSize < threshold.
Can you add it as config in Gluten?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499405185
> > The kSpillReadBufferSize controls the size to read each file, the ordered reader will create FileOutputStream for each file and allocate kSpillReadBufferSize (default 1MB) for one file. Can you try to adjust this value? Maybe we should add a new config to control the read buffer size for all the files. numFiles * bufferSize < threshold.
>
> should we propose the way of #7861?
#7861 releases the buffer after read, Velox FileInputStream reuse the `readBufferSize` to read the file. So it is the similar way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2495369792
I can only find the configuration: spark.shuffle.spill.diskWriteBufferSize.
No spill merge one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499590766
> s, I will draft a PR to respect
I'm adding to #8026
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2496548266
In that case, we need to respect UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE.
Velox spill RowVector, so we must read the `buffer size` ahead. But it's better to add a config for total buffer size when UNSAFE_SORTER_SPILL_READ_AHEAD_ENABLED is false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499588141
> #7861 releases the buffer after read, Velox FileInputStream reuse the `readBufferSize` to read the file. So it is the similar way.
No, 7681 uses mmap, memory is mapped into user space directly. velox uses file read/write, data is copied to buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499760082
Now it is Spark codec `spark.io.compression.codec`, I add the config to control it separately.
PrestoVectorSerde has Header, so it should read the compressed data and deserialize to RowVector. And hold RowVector for each file. The many RowVectors may cause OOM.
In SortBuffer `getOutputWithSpill`, the `std::vector<const RowVector*> spillSources_;` and `std::vector<vector_size_t> spillSourceRows_;` size is equal to outputSize, which also holds memory but not tracked, may cause Kill By Yarn. I will draft a PR to track the memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2496510231
The kSpillReadBufferSize controls the size to read each file, the ordered reader will create FileOutputStream for each file and allocate kSpillReadBufferSize (default 1MB) for one file. Can you try to adjust this value?
Maybe we should add a new config to control the read buffer size for all the files. numFiles * bufferSize < threshold.
```
for (auto& fileInfo : files_) {
streams.push_back(FileSpillMergeStream::create(
SpillReadFile::create(fileInfo, bufferSize, pool, spillStats)));
}
input_ = std::make_unique<common::FileInputStream>(
std::move(file), bufferSize, pool_);
FileInputStream::FileInputStream(
std::unique_ptr<ReadFile>&& file,
uint64_t bufferSize,
memory::MemoryPool* pool)
: file_(std::move(file)),
fileSize_(file_->size()),
bufferSize_(std::min(fileSize_, bufferSize)),
pool_(pool),
readAheadEnabled_((bufferSize_ < fileSize_) && file_->hasPreadvAsync()) {
VELOX_CHECK_NOT_NULL(pool_);
VELOX_CHECK_GT(fileSize_, 0, "Empty FileInputStream");
buffers_.push_back(AlignedBuffer::allocate<char>(bufferSize_, pool_)); // allocate buffer cause OOM
if (readAheadEnabled_) {
buffers_.push_back(AlignedBuffer::allocate<char>(bufferSize_, pool_));
}
readNextRange();
}
```
kSpillWriteBufferSize controls the serialization buffer, if up to this threshold, flush and compress the buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499462328
> Thank you, @jinchengchenghh . With the tuning of kMaxSpillRunRows and kSpillWriteBufferSize. one of my task succeed but the other one still fails. Looks like it still have some large memory allocation in getoutput.
Maybe because the Streams will hold all the buffers, and released after all the files read completed.
Compress consumes much buffer but not tracked by memory pool in the meantime. https://github.com/facebookincubator/velox/blob/main/velox/serializers/PrestoSerializer.cpp#L4416
I don't see the compression in Spark spill, so it doesn't need to request memory for compression.
I will add a new config to control the velox spill codec.
It is still OOM or kill by yarn?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2495309752
Looks it's the same issue as shuffle spill. All spill merge should have the same issue. we should solve it by similar way.
1) is the tradeoff of spill file# and overhead memory.
2) I'd think 4M is too large.
What's the vanilla spark's spill buffer size? is it configurable? in theory vanilla spark has the same issue as Gluten. @jinchengchenghh do you know?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499339837
> In that case, we need to respect UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE. Velox spill RowVector, so we must read the `buffer size` ahead. But it's better to add a config for total buffer size when UNSAFE_SORTER_SPILL_READ_AHEAD_ENABLED is false.
So the worst case of Vanilla spark is also 1M buffer per file, right? Let's hornor the value of spark.unsafe.sorter.spill.reader.buffer.size then. It may be set in queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499592257
> I will add a new config to control the velox spill codec.
>
> It is still OOM or kill by yarn?
we already should have spill codec and spill buffer configured.
OOM, not killed by yarn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2499549316
> UNSAFE_SORTER_SPILL_READER_BUFFER_SIZE
And also close the serializer in serde.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on issue #8025:
URL: https://github.com/apache/incubator-gluten/issues/8025#issuecomment-2510427960
Could you try to set `spark.gluten.sql.columnar.backend.velox.spillStartPartitionBit = 0 spark.gluten.sql.columnar.backend.velox.spillPartitionBits = 0`? Velox hash the rows and then write to each file by pid in case of parallel for Presto Velox, https://github.com/facebookincubator/velox/blob/main/velox/exec/Spiller.cpp#L701, so it may generate many small files while Spark does not hash.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Distinct aggregation OOM when getOutput [gluten]
Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on issue #8025:
URL: https://github.com/apache/gluten/issues/8025#issuecomment-4073134619
> Could you try to set `spark.gluten.sql.columnar.backend.velox.spillStartPartitionBit = 0 spark.gluten.sql.columnar.backend.velox.spillPartitionBits = 0`? Velox hash the rows and then write to each file by pid in case of parallel for Presto Velox, https://github.com/facebookincubator/velox/blob/main/velox/exec/Spiller.cpp#L701, so it may generate many small files while Spark does not hash.
@jinchengchenghh Hi, is it suitable to set this default to 0 for Spark, is there any side effect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org