You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "guowangy (via GitHub)" <gi...@apache.org> on 2026/03/09 02:54:23 UTC
[PR] [VL] Support multiple segments per partition in columnar shuffle [incubator-gluten]
guowangy opened a new pull request, #11722:
URL: https://github.com/apache/incubator-gluten/pull/11722
## What changes are proposed in this pull request?
Introduces multi-segment-per-partition support in the Velox backend columnar shuffle writer, enabling incremental flushing of partition data to the final data file during processing — reducing peak memory usage without requiring full in-memory buffering or temporary spill files. The implementation can reduce total latency of TPC-H(SF6T) by ~16% using sort-based shuffle with low memory capacity in 2-socket Xeon 6960P system.
### New index file format (`ColumnarIndexShuffleBlockResolver`)
Extends `IndexShuffleBlockResolver` with a new index format supporting multiple `(offset, length)` segments per partition:
```
[Partition Index: (N+1) × 8-byte big-endian offsets]
[Segment Data: per-partition list of (data_offset, length) pairs, each 8 bytes]
[1-byte end marker] ← distinguishes from legacy format (size always multiple of 8)
```
<img width="1013" height="381" alt="image" src="https://github.com/user-attachments/assets/74e278e1-3d8e-4225-95a6-7c9eeac133ce" />
`ColumnarShuffleManager` now uses this resolver. Multi-segment mode activates only when external shuffle service, push-based shuffle, and dictionary encoding are all disabled (dictionary encoding requires all-batches-complete before writing).
### New I/O abstractions
- **`FileSegmentsInputStream`** — `InputStream` over non-contiguous `(offset, size)` file segments; supports zero-copy native reads via `read(destAddress, maxSize)`
- **`FileSegmentsManagedBuffer`** — `ManagedBuffer` backed by discontiguous segments; supports `nioByteBuffer()`, `createInputStream()`, `convertToNetty()`
- **`DiscontiguousFileRegion`** — Netty `FileRegion` mapping a logical range to multiple physical segments for zero-copy network transfer
- **`LowCopyFileSegmentsJniByteInputStream`** — zero-copy JNI wrapper over `FileSegmentsInputStream`; wired into `JniByteInputStreams.create()`
### C++ `LocalPartitionWriter` changes
- `usePartitionMultipleSegments_` flag + `partitionSegments_` vector tracking `(start, length)` per partition
- `flushCachedPayloads()` — incremental flush after each `hashEvict`
- `writeMemoryPayload()` — direct write to final data file during `sortEvict`
- `writeIndexFile()` — serializes the new index at stop time
- `PayloadCache::writeIncremental()` — flushes completed (non-active) partitions without touching the in-use partition
### JNI/JVM wiring
`LocalPartitionWriterJniWrapper` and `JniWrapper.cc` accept a new optional `indexFile` parameter; `ColumnarShuffleWriter` passes the temp index file path when multi-segment mode is active.
## How was this patch tested?
New unit test suites:
- `ColumnarIndexShuffleBlockResolverSuite` — index format read/write, format detection, multi-segment block lookup
- `FileSegmentsInputStreamSuite` — sequential reads, multi-segment traversal, skip, zero-copy native reads
- `FileSegmentsManagedBufferSuite` — `nioByteBuffer`, `createInputStream`, `convertToNetty`, EOF and mmap edge cases
- `DiscontiguousFileRegionSuite` — Netty transfer across discontiguous segments, lazy open
- `LowCopyFileSegmentsJniByteInputStreamTest` — JNI wrapper correctness for ByteInputStream
## Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4218703771
> @guowangy In general, random I/O is considered a bottleneck in shuffle, and that's why there are so many remote shuffle service projects and solutions like celeborn, uniffle are aimed at. The remote shuffle service usually coalesce the shuffle outputs from mapper side to reduce the random IO access. However, the design in this PR seems to go in the opposite direction, since it may introduce more random I/O during reads.
>
> Directly writing the segments to the data file would make the partition writer logic simpler, but we intentionally didn't choose that approach based on the above consideration. I'm not sure if your test is based on single node or on a cluster. If it's on single node and disk IO is not bottleneck, then the solution may not be practical in real use case.
>
> Besides, based on our experience, external shuffle service is usually enabled in real production environments because it provides better stability when executor process is down, and it's more like a must-have feature that the shuffle framework should support.
make sense to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4037658664
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #11722: [VL] Support multiple segments per partition in columnar shuffle
URL: https://github.com/apache/gluten/pull/11722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4034719163
> The implementation can reduce total latency of TPC-H(SF6T) by ~16% using sort-based shuffle with low memory capacity in 2-socket Xeon 6960P system.
Can you explain where this improvement mainly comes from?
Currently we follow the same file layout as vanilla spark to have each partition output contiguous. I think one major benefit for this design is to reduce small random disk IO from the shuffle reader side. If memory is tight then the spill will be triggered more frequently, and it will be more likely to produce small output blocks for each partition. In this case this design will not be IO friendly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4037894211
@guowangy In general, random I/O is considered a bottleneck in shuffle, and that's why there are so many remote shuffle service projects and solutions like celeborn, uniffle are aimed at. The remote shuffle service usually coalesce the shuffle outputs from mapper side to reduce the random IO access. However, the design in this PR seems to go in the opposite direction, since it may introduce more random I/O during reads.
Directly writing the segments to the data file would make the partition writer logic simpler, but we intentionally didn't choose that approach based on the above consideration. I'm not sure if your test is based on single node or on a cluster. If it's on single node and disk IO is not bottleneck, then the solution may not be practical in real use case.
Besides, based on our experience, external shuffle service is usually enabled in real production environments because it provides better stability when executor process is down, and it's more like a must-have feature that the shuffle framework should support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4037681254
@guowangy I draw a graph based on my understanding. Please correct me if I'm wrong: The existing design only has one random I/O access to one mapper output per reducer, but the new design has more random I/O accesses when reading all the segments from one mapper per reducer.
<img width="491" height="703" alt="image" src="https://github.com/user-attachments/assets/66399c0e-8acd-43fc-ba2c-31671692aaac" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on code in PR #11722:
URL: https://github.com/apache/gluten/pull/11722#discussion_r2913163731
##########
cpp/core/shuffle/LocalPartitionWriter.cc:
##########
@@ -562,6 +583,56 @@ void LocalPartitionWriter::init() {
std::default_random_engine engine(rd());
std::shuffle(localDirs_.begin(), localDirs_.end(), engine);
subDirSelection_.assign(localDirs_.size(), 0);
+
+ if (!indexFile_.empty()) {
+ usePartitionMultipleSegments_ = true;
+ partitionSegments_.resize(numPartitions_);
+ }
+}
+
+// Helper for big-endian conversion (network order)
+#include <arpa/inet.h>
+static uint64_t htonll(uint64_t value) {
+#if __BYTE_ORDER == __LITTLE_ENDIAN
+ return (((uint64_t)htonl(value & 0xFFFFFFFFULL)) << 32) | htonl(value >> 32);
+#else
+ return value;
+#endif
+}
+
+arrow::Status LocalPartitionWriter::writeIndexFile() {
Review Comment:
Can you add some c++ unit tests for the multi-segment partition write?
##########
backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala:
##########
@@ -52,6 +55,18 @@ class ColumnarShuffleWriter[K, V](
s"expected one of: ${HashShuffleWriterType.name}, ${SortShuffleWriterType.name}")
}
+ shuffleBlockResolver match {
+ case resolver: ColumnarIndexShuffleBlockResolver =>
+ if (resolver.canUseNewFormat() && !GlutenConfig.get.columnarShuffleEnableDictionary) {
+ // For Dictionary encoding, the dict only finalizes after all batches are processed,
+ // and dict is required to saved at the head of the partition data.
+ // So we cannot use multiple segments to save partition data incrementally.
+ partitionUseMultipleSegments = true
Review Comment:
Please add a configuration to enable this feature.
##########
cpp/core/shuffle/LocalPartitionWriter.cc:
##########
@@ -759,6 +881,12 @@ arrow::Status
LocalPartitionWriter::sortEvict(uint32_t partitionId, std::unique_ptr<InMemoryPayload> inMemoryPayload, bool isFinal, int64_t& evictBytes) {
rawPartitionLengths_[partitionId] += inMemoryPayload->rawSize();
+ if (usePartitionMultipleSegments_) {
+ // If multiple segments per partition is enabled, write directly to the final data file.
+ RETURN_NOT_OK(writeMemoryPayload(partitionId, std::move(inMemoryPayload)));
Review Comment:
Can you explain a bit more on how this can reduce the memory usage? Looks like the memory is still only get reclaimed by OOM and spilling.
##########
cpp/core/shuffle/LocalPartitionWriter.cc:
##########
@@ -562,6 +583,56 @@ void LocalPartitionWriter::init() {
std::default_random_engine engine(rd());
std::shuffle(localDirs_.begin(), localDirs_.end(), engine);
subDirSelection_.assign(localDirs_.size(), 0);
+
+ if (!indexFile_.empty()) {
+ usePartitionMultipleSegments_ = true;
+ partitionSegments_.resize(numPartitions_);
+ }
+}
+
+// Helper for big-endian conversion (network order)
+#include <arpa/inet.h>
Review Comment:
Please move the header to the top, and move `htoll` into anonymous namespace after the headers.
##########
cpp/core/shuffle/LocalPartitionWriter.cc:
##########
@@ -750,6 +869,9 @@ arrow::Status LocalPartitionWriter::hashEvict(
for (auto& payload : merged) {
RETURN_NOT_OK(payloadCache_->cache(partitionId, std::move(payload)));
}
+ if (usePartitionMultipleSegments_) {
+ RETURN_NOT_OK(flushCachedPayloads());
Review Comment:
The hashEvict is not only called for spilling. When the evictType is `kCache`, then it try to cache as much payload in memory as possible to reduce spilling.
And when the evitType is `kSpill`, the data will be written to a spilled data file. Two evict types can exist in the same job. Is `evictType == kSpill` being properly handled for multi-segments write?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4618370160
This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4037909401
cc: @FelixYBW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "guowangy (via GitHub)" <gi...@apache.org>.
guowangy commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4037707027
> @guowangy I draw a graph based on my understanding. Please correct me if I'm wrong: The existing design only has one random I/O access to one mapper output per reducer, but the new design has more random I/O accesses when reading all the segments from one mapper per reducer.
>
> <img alt="image" width="491" height="703" src="https://private-user-images.githubusercontent.com/52736607/561469832-66399c0e-8acd-43fc-ba2c-31671692aaac.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzMyMjA5ODcsIm5iZiI6MTc3MzIyMDY4NywicGF0aCI6Ii81MjczNjYwNy81NjE0Njk4MzItNjYzOTljMGUtOGFjZC00M2ZjLWJhMmMtMzE2NzE2OTJhYWFjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzExVDA5MTgwN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBiZmRkYWZmNjBmYTZlOGFiNjkxMjY0NGRmMjk3YWU2ODdiYTM3ZDY4ZmNiYzBhNzhiNjlkYzFlMzhhM2MwYmEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.uO-S8Fc0CMFjOUbLvmj5CAYsWPO7LTcvv8a3WnovDoU">
Yes, for shuffle reader, that's true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "guowangy (via GitHub)" <gi...@apache.org>.
guowangy commented on code in PR #11722:
URL: https://github.com/apache/gluten/pull/11722#discussion_r2915546898
##########
cpp/core/shuffle/LocalPartitionWriter.cc:
##########
@@ -759,6 +881,12 @@ arrow::Status
LocalPartitionWriter::sortEvict(uint32_t partitionId, std::unique_ptr<InMemoryPayload> inMemoryPayload, bool isFinal, int64_t& evictBytes) {
rawPartitionLengths_[partitionId] += inMemoryPayload->rawSize();
+ if (usePartitionMultipleSegments_) {
+ // If multiple segments per partition is enabled, write directly to the final data file.
+ RETURN_NOT_OK(writeMemoryPayload(partitionId, std::move(inMemoryPayload)));
Review Comment:
Reduce memory usage does not apply to sortEvict since it is usually triggered at spill.
But it is applicable for hashEvict because payloads don't need cache to memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4531020346
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11722:
URL: https://github.com/apache/incubator-gluten/pull/11722#issuecomment-4020758091
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "guowangy (via GitHub)" <gi...@apache.org>.
guowangy commented on PR #11722:
URL: https://github.com/apache/gluten/pull/11722#issuecomment-4035805472
> > The implementation can reduce total latency of TPC-H(SF6T) by ~16% using sort-based shuffle with low memory capacity in 2-socket Xeon 6960P system.
>
> Can you explain where this improvement mainly comes from?
>
> Currently we follow the same file layout as vanilla spark to have each partition output contiguous. I think one major benefit for this design is to reduce small random disk IO from the shuffle reader side. If memory is tight then the spill will be triggered more frequently, and it will be more likely to produce small output blocks for each partition. In this case this design will not be IO friendly.
For existing design, if spill happens, interim blocks of partition data will be saved to the disk; at the end of the shuffle write, these interim data will be reloaded from disk and saved as final partition data. This optimization is trying to avoid such interim write/reload operations, when spill happens, data will be directly saved as final partition blocks.
As for IO friendly, for existing design, if spill is triggered frequently, there will be also many small interim blocks saved to disk, it's not IO friendly when reload them from disk and packed as final partition data. With this PR, it does not make IO friendly worse, just move such situation from shuffle writer to reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Support multiple segments per partition in columnar shuffle [gluten]
Posted by "guowangy (via GitHub)" <gi...@apache.org>.
guowangy commented on code in PR #11722:
URL: https://github.com/apache/gluten/pull/11722#discussion_r2915560966
##########
cpp/core/shuffle/LocalPartitionWriter.cc:
##########
@@ -750,6 +869,9 @@ arrow::Status LocalPartitionWriter::hashEvict(
for (auto& payload : merged) {
RETURN_NOT_OK(payloadCache_->cache(partitionId, std::move(payload)));
}
+ if (usePartitionMultipleSegments_) {
+ RETURN_NOT_OK(flushCachedPayloads());
Review Comment:
Yes, `evictType == kSpill` is properly handled.
Because when `usePartitionMultipleSegments_` enabled, the cache mechanism of `payloadCache_` is not used. Payloads will directly flush into disk; thus, we don't need to distinguish between `kCache` and `kSpill`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org