You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/12 19:52:20 UTC
[GitHub] [spark] zhengruifeng opened a new pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
zhengruifeng opened a new pull request #33298:
URL: https://github.com/apache/spark/pull/33298
### What changes were proposed in this pull request?
1, introduce `ShuffleExecAccumulator` in `ShuffleExchangeExec` to support arbitrary statistics;
2, impl a key sampling `ShuffleExecAccumulator` to detect skew keys and show debug info on SparkUI;
3, in `OptimizeSkewedJoin`, estimate the joined size of each partition based on the sampled keys, and split a partition if it is not split yet and its estimated joined size is too larger.
### Why are the changes needed?
1, make it easy to add a use-defined statistics which can be used in AQE rules;
2, showing skew info on sparkUI is usefully;
3, spliting partitions based on joined size can resolve data inflation;
### Does this PR introduce _any_ user-facing change?
Yes, new features are added
### How was this patch tested?
added testsuites
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-937377918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-937377918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-937374133
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-878197315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-878125294
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-878197315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-937377918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-937374133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-937376825
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #33298:
URL: https://github.com/apache/spark/pull/33298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-878125294
**[Test build #140910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140910/testReport)** for PR 33298 at commit [`a9888ba`](https://github.com/apache/spark/commit/a9888ba9458d012ab71644bc0808e8123955a824).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33298: [SPARK-36087][SQL][WIP] An Impl of skew key detection and data inflation optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33298:
URL: https://github.com/apache/spark/pull/33298#issuecomment-937377918
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org