You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "FelixYBW (via GitHub)" <gi...@apache.org> on 2024/08/01 18:43:58 UTC
[I] [VL] Difference of Spill between Vanilla and Gluten [incubator-gluten]
FelixYBW opened a new issue, #6683:
URL: https://github.com/apache/incubator-gluten/issues/6683
### Description
Here is a summary of difference of Spill policy between Vanilla and Gluten:
• All vanilla spark SQL operators spill support is through external sort
• External sort only spill itself
• External sort will hold all data when it's called
• The spill API in MemoryConsumer
○ Not useful in SparkSQL
• Gluten may spill data before data are cached
○ Hashjoin (already spilled) + sort (parquet write)
○ PR #6480 is essential to Gluten now
• Assume a Sort+sort
○ vanilla sort + vanilla spark: Vanilla spark always spill the 2nd sort
§ PR6480 will break this
○ If both sort are in velox, which one is spilled is decided by Velox
○ Gluten sort + vanilla spark sort. 2nd sort will spill Gluten's sort
○ Vanilla sort + Gluten sort. 2nd sort can't spill Vanilla's sort without PR6480
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Difference of Spill between Vanilla and Gluten [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6683:
URL: https://github.com/apache/incubator-gluten/issues/6683#issuecomment-2277235905
In Velox, if task A is going to spill task B's data, task B will be paused until spill finish. But in Vanilla Spark, it's not the case. The target task isn't paused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Difference of Spill between Vanilla and Gluten [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #6683:
URL: https://github.com/apache/incubator-gluten/issues/6683#issuecomment-2266326956
#6480
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org