You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "touchida (via GitHub)" <gi...@apache.org> on 2023/07/16 14:09:31 UTC
[GitHub] [kyuubi] touchida opened a new issue, #5057: [Improvement] Support local sort in the Spark's insertRepartitionBeforeWrite optimization rule
touchida opened a new issue, #5057:
URL: https://github.com/apache/kyuubi/issues/5057
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
Currently, the Spark's `insertRepartitionBeforeWrite` optimization rule will be skipped when logical plans are `Sort` regardless of whether they are local or not: https://github.com/apache/kyuubi/blob/fa9e6be/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/RepartitionBeforeWritingBase.scala#L133.
It makes sense for global sort, since inserting repartition after the sort changes the semantics of the original plans and doing before that only introduces an additional shuffle.
However, inserting repartition before local sort will help to sort rebalanced partitions even if locally, and it follows the behaviors of queries that explicitly use both `REPARTITION|REBALANCE` and `SORT BY`.
### How should we improve?
This issue proposes to support local sort in the Spark's `insertRepartitionBeforeWrite` optimization rule by inserting repartition before the sort.
### Are you willing to submit PR?
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #5057: [Improvement] Support local sort in the Spark's insertRepartitionBeforeWrite optimization rule
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #5057:
URL: https://github.com/apache/kyuubi/issues/5057#issuecomment-1637281973
cc @ulysses-you @cxzl25 @cfmcgrady what do you think about this idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org