You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "j7nhai (via GitHub)" <gi...@apache.org> on 2025/03/28 03:06:25 UTC
[I] Shuffle write data size is larger for sort-based columnar shuffle [incubator-gluten]
j7nhai opened a new issue, #9163:
URL: https://github.com/apache/incubator-gluten/issues/9163
### Backend
VL (Velox)
### Bug description
We found that hash columnar shuffle performed poorly and generated too many disk files.
So we used **sort columnar shuffle**, but it still generated more shuffle write data than vanilla spark in the same stage. For example, for the same stage, the performance is as follows:
gluten:
```
Resource Profile Id: 0
Total Time Across All Tasks: 123.4 h
Locality Level Summary: Process local: 10604
Input Size / Records: 4.3 TiB / 24137351937
Shuffle Write Size / Records: 4.0 TiB / 24137134849
Spill (Memory): 39.8 TiB
Spill (Disk): 3.5 TiB
Associated Job Ids: 0
```
vanilla spark:
```
Resource Profile Id: 0
Total Time Across All Tasks: 102.0 h
Locality Level Summary: Process local: 10492
Input Size / Records: 4.0 TiB / 24083129089
Shuffle Write Size / Records: 2.6 TiB / 24083129089
Spill (Memory): 42.5 TiB
Spill (Disk): 2.2 TiB
Associated Job Ids: 0
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Shuffle write data size is larger for sort-based columnar shuffle [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #9163:
URL: https://github.com/apache/incubator-gluten/issues/9163#issuecomment-2830864517
@j7nhai Could you help to check if the issue is fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Shuffle write data size is larger for sort-based columnar shuffle [incubator-gluten]
Posted by "M1zukiqwq (via GitHub)" <gi...@apache.org>.
M1zukiqwq commented on issue #9163:
URL: https://github.com/apache/incubator-gluten/issues/9163#issuecomment-3088362137
@j7nhai hi, i want to know how do you set to use sort-based shuffle, did you set it by spark.gluten.sql.columnar.shuffle.sort.threshold?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Shuffle write data size is larger for sort-based columnar shuffle [incubator-gluten]
Posted by "j7nhai (via GitHub)" <gi...@apache.org>.
j7nhai closed issue #9163: [VL] Shuffle write data size is larger for sort-based columnar shuffle
URL: https://github.com/apache/incubator-gluten/issues/9163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Shuffle write data size is larger for sort-based columnar shuffle [incubator-gluten]
Posted by "j7nhai (via GitHub)" <gi...@apache.org>.
j7nhai commented on issue #9163:
URL: https://github.com/apache/incubator-gluten/issues/9163#issuecomment-2833322788
This issue is fixed now 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Shuffle write data size is larger for sort-based columnar shuffle [incubator-gluten]
Posted by "marin-ma (via GitHub)" <gi...@apache.org>.
marin-ma commented on issue #9163:
URL: https://github.com/apache/incubator-gluten/issues/9163#issuecomment-3088396350
@M1zukiqwq You can set either `spark.gluten.sql.columnar.shuffle.sort.partitions.threshold` or `spark.gluten.sql.columnar.shuffle.sort.columns.threshold`
Please check https://github.com/apache/incubator-gluten/blob/main/docs/Configuration.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org