You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "NEUpanning (via GitHub)" <gi...@apache.org> on 2025/07/02 09:56:00 UTC
[I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
NEUpanning opened a new issue, #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104
### Backend
VL (Velox)
### Bug description
The total time across all tasks for vanilla Spark was 78.1 hours, but for gluten it reached **1899.3** hours. The flame graph shows that the majority of time is occupied by the merging of payloads. After adding some logs, I see merge operation occurred 1522330 times for 1084128 rows in a task, with each instance taking x milliseconds.
Flame graph:
<img width="1202" alt="Image" src="https://github.com/user-attachments/assets/3ec70fb5-52c5-45c3-afc9-163fdd36a026" />
gluten shuffle metrics:
```
shuffle records written: 39,407,858,231
shuffle write time total (min, med, max (stageId: taskId))
17.86 h (0 ms, 27.4 s, 17.1 m (stage 0.0: task 499))
time to compress total (min, med, max (stageId: taskId))
31.52 h (0 ms, 29.6 s, 20.3 m (stage 0.0: task 1191))
time to split total (min, med, max (stageId: taskId))
1478.09 h (0 ms, 45.5 m, 1.71 h (stage 0.0: task 86))
time to spill total (min, med, max (stageId: taskId))
15.77 h (0 ms, 25.2 s, 16.3 m (stage 0.0: task 499))
```
shuffle schema:
<img width="203" alt="Image" src="https://github.com/user-attachments/assets/c3466aee-b5d3-46b0-80db-5b208ac393eb" />
### Gluten version
Gluten-1.3
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3051321950
@NEUpanning Is your data in the pattern that only 1 or a few reducer partitions are filled during the split?
Here we allocate the destination row vector size by available memory/reducer numbers, the assumption is that the data is evenly filled into destination partitions. But if the data is somehow sorted or skewed, the performance will be bad.
The solution is to allocate the vector size by the more precised reducer number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3035667687
@NEUpanning Could you share what's the previous operator before shuffle? Is it a sort operator or the input data is already sorted ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3027235859
cc @marin-ma
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3035823111
> We can support preallocating the merged buffer size to reduce the number of resizing.
@marin-ma I tried to support this feature for POC, but there is no performance gain for this case. Could you help to take a look if the approach is correct? Thanks! Here is the code: https://github.com/apache/incubator-gluten/commit/bf3c4e1fa256219bce2570086b20cdcd491ed85d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3109962019
> This issue has been resolved by using rss sort-based shuffle and this case only costs 74% of the cost time of vanilla version. Thank you so much for your help! @FelixYBW @marin-ma
@NEUpanning Curious are you using RSS or not? The rss sort shuffle is only used for celeborn, but the flamegraph in the issue description shows it was not using rss for hash based shuffle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3120358478
> Could you explain what PSS is?
@marin-ma Spark push-based shuffle service which means we set spark.shuffle.push.enabled to true.
> Have you try the sort based shuffle without RSS?
I tested it too. Sort based shuffle without RSS still does not have a better performance compared to vanilla Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3034736562
It's the memory allocation issue. @marin-ma do we pre-allocate memory for merge of batches? If not we need to add this optimizations. When we do the merge, do we copy the string itself or the string pointer in stringview?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3113435135
> This case occurred in hash based shuffle with RSS.
It shouldn't be. The hash based shuffle with RSS doesn't have the "merge payload" logic. It directly sends the payloads to the RSS through the client.
The flamegraph you shared is from LocalPartitionWriter, and it's not using RSS.
<img width="2404" height="1400" alt="Image" src="https://github.com/user-attachments/assets/5a992821-0bca-4a9f-8b93-dee187a46875" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3035734699
> Could you share what's the previous operator before shuffle? Is it a sort operator or the input data is already sorted ?
@marin-ma There is no sort op before shuffle.
<img width="437" alt="Image" src="https://github.com/user-attachments/assets/674d53e3-323c-4344-9581-c30ced9ae812" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3027467974
@marin-ma
> Could you share other metrics for the ColumnarShuffleExchange operator?
```
ColumnarExchange
shuffle records written: 39,407,858,231
shuffle write time total (min, med, max (stageId: taskId))
17.86 h (0 ms, 27.4 s, 17.1 m (stage 0.0: task 499))
time to compress total (min, med, max (stageId: taskId))
31.52 h (0 ms, 29.6 s, 20.3 m (stage 0.0: task 1191))
time to split total (min, med, max (stageId: taskId))
1478.09 h (0 ms, 45.5 m, 1.71 h (stage 0.0: task 86))
shuffle wall time total (min, med, max (stageId: taskId))
1527.48 h (0 ms, 46.5 m, 1.92 h (stage 0.0: task 86))
number of input rows: 39,407,858,231
time to spill total (min, med, max (stageId: taskId))
15.77 h (0 ms, 25.2 s, 16.3 m (stage 0.0: task 499))
shuffle bytes spilled total (min, med, max (stageId: taskId))
3.0 TiB (0.0 B, 1516.5 MiB, 1944.8 MiB (stage 0.0: task 909))
number of input batches: 9,788,389
Native.mergeSpillsTime total (min, med, max (stageId: taskId))
34.92 h (0 ms, 33.8 s, 23.1 m (stage 0.0: task 1191))
data size total (min, med, max (stageId: taskId))
75.6 TiB (0.0 B, 38.8 GiB, 38.9 GiB (stage 0.0: task 1))
peak bytes allocated total (min, med, max (stageId: taskId))
7.5 TiB (0.0 B, 5.1 GiB, 5.6 GiB (stage 0.0: task 1418))
number of partitions: 2,000
shuffle bytes written total (min, med, max (stageId: taskId))
3.7 TiB (0.0 B, 1944.5 MiB, 1953.5 MiB (stage 0.0: task 1))
```
> You may also try enabling sort-based shuffle and see if it can get better performance.
The number of shuffle partitions is only 2000, which is not greater than `spark.gluten.sql.columnar.shuffle.sort.partitions.threshold`. I'll try to force enable it.
And I adjusted merge buffer size from 4096 to 2048, resulting in a ~30-fold reduction in split time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3111735191
> Curious are you using RSS or not?
@marin-ma This case occurred in hash based shuffle with PSS. Using RSS with celeborn, the ETL does not have this performance issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3027670661
@NEUpanning `spark.gluten.sql.columnar.shuffle.merge.threshold` default 0.25, meaning any buffer < 0.25 * 4096 will be saved for merging. You may set it to a lower value or set to 0 to disable merging them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3101327567
This issue has been resolved by using rss sort-based shuffle and this case only costs 74% of the cost time of vanilla version. Thank you so much for your help! @FelixYBW @marin-ma
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3035299119
@FelixYBW The merged ones are not using pre-allocated memory, and all data are copied (not using StirngView in arrow buffer)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3034660597
> what's the performance when you disable the batch merge?
I've not tried to disable the batch merge, but I adjusted merge buffer size from 4096 to 2048 and the split time decreased by ~30 times.
> What's the column type of your shuffle data?
There are 4 string columns and 1 int column.
@FelixYBW
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3035314730
@NEUpanning We can support preallocating the merged buffer size to reduce the number of resizing. If the split buffer is too small but the final merged buffer is large, reducing the number of shuffle partitions should help. Could you try reducing the `spark.sql.shuffle.partitions`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3027528073
Additionally, I am wondering if we could increase the capacity of [cached merge payload](https://github.com/apache/incubator-gluten/blob/main/cpp/core/shuffle/LocalPartitionWriter.cc#L218) buffers in advance to avoid resizing them multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3045629870
> [@marin-ma](https://github.com/marin-ma) I tried to support this feature for POC, but there is no performance gain for this case. Could you help to take a look if the approach is correct? Thanks! Here is the code: [bf3c4e1](https://github.com/apache/incubator-gluten/commit/bf3c4e1fa256219bce2570086b20cdcd491ed85d)
@NEUpanning Seems like this change doesn't reduce the number of resizing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3116598720
@NEUpanning I thought that was a typo. Could you explain what PSS is? Have you try the sort based shuffle without RSS?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3107660659
> Could you check if your data is pre sorted or skewed?
@FelixYBW I see the data is not skewed, with the number of rows in each partition is ranging 9500 to 10500. And I notice the number of appended rows is very small, the tp95 is only 9 rows. Hope this could help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3116116747
@marin-ma
For some reason, the comment you quoted does **not** match with my comment which is without any editing.
<img width="977" height="398" alt="Image" src="https://github.com/user-attachments/assets/bf5bd021-a391-4d65-a40d-125d6df65106" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3027241334
Could you share other metrics for the ColumnarShuffleExchange operator? You may also try enabling sort-based shuffle can see if can get better performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "NEUpanning (via GitHub)" <gi...@apache.org>.
NEUpanning commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3034351501
> You may set it to a lower value or set to 0 to disable merging them.
@marin-ma I see 95% appended payloads contain fewer than 10 rows in this case, so only disabling merging through this configuration can solve it. But I assume disabling merging could result in performance regression for other ETLs. Can we make the merge buffer size configurable or optimize the merge process like [I mentioned above](https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3027528073)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3034450951
what's the performance when you disable the batch merge?
The merge itself is to solve the small batch issue because we'd expect the split of small batches have big overhead.
What's the column type of your shuffle data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] The time taken to merge payload during the shuffle write is excessively high [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #10104:
URL: https://github.com/apache/incubator-gluten/issues/10104#issuecomment-3101879629
Thank you for your update, @NEUpanning . Actually we are still not sure how the small bach size is allocated. Could you check if your data is pre sorted or skewed? would you mind to help us print some debug info? which can help to improve current hash based solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org