You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lianhuiwang <gi...@git.apache.org> on 2015/01/16 13:43:37 UTC
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
GitHub user lianhuiwang opened a pull request:
https://github.com/apache/spark/pull/4070
[SPARK-4630][Core]Dynamically determine optimal number of partitions
stages in application have different size of data. if user doesnot set numPartitions for any stages, spark will use same defaultParallelism as partitons.
then in DAGScheduler number of stage's running tasks is equal to partition size of stage.so usually this number is a same value.
so if number of stage's partitions is too small, then task need to process large mount of data and slows down due to spilling or gc.
if number of stage's partitions is too large, there is big cost in schedule.
To improve performance of application, we need to determine optimal number of partitions according to stage's input data size.
there are two steps:
1. estimate number of Stage's Partitions
according to stage's parent stages's input data size and spark.reduce.per.partition.bytes configuration we can determine number of stage's partitions.
how to get parent stages's input data Size?
if it has no parent, we get its inputSize through accumulating its input path's length.
else if it has parents, but its parents is no available, so we get its parents' input data size.
else if its parents is avilable, we just compute its parents' shuffle data size as its input size.
2. update Stage's Partitioner
firstly, update partition of parents' shuffleDep. that make shuffleMapTask write wanted number of partition files.
and then, update stage's information, particularly stage's rdd. it make stage's shuffleRDD can correctly pull data from map task.
TODO:
1. consider spark.shuffle.memoryFraction to determine spark.reduce.per.partition.bytes configuration.
2. when stage is final stage, resultHandler's value cannot be returned to SparkContext because partitions has been changed.
3. when number of stage's tasks has been changed, report stage's new information to UI.
@ksakellis @sryza @JoshRosen @rxin
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lianhuiwang/spark SPARK-4630
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4070.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4070
----
commit fc652a5aefe4d694d0c3374dce6b3c4415c7be56
Author: lianhuiwang <li...@gmail.com>
Date: 2015-01-16T12:38:03Z
Dynamically determine optimal number of partitions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70274053
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25662/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70249150
[Test build #25658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25658/consoleFull) for PR 4070 at commit [`fc652a5`](https://github.com/apache/spark/commit/fc652a5aefe4d694d0c3374dce6b3c4415c7be56).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HashPartitioner(var partitions: Int) extends Partitioner `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70461533
@lianhuiwang have you tried this in production on any jobs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70250149
[Test build #25659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25659/consoleFull) for PR 4070 at commit [`668926c`](https://github.com/apache/spark/commit/668926cc681320b1d2fec191c9e7917384dc4803).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HashPartitioner(var partitions: Int) extends Partitioner `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70258988
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25660/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70274042
[Test build #25662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25662/consoleFull) for PR 4070 at commit [`622e45c`](https://github.com/apache/spark/commit/622e45c2bebea7a6206fbf29569e766f57a7f17b).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HashPartitioner(var partitions: Int) extends Partitioner `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70249983
[Test build #25659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25659/consoleFull) for PR 4070 at commit [`668926c`](https://github.com/apache/spark/commit/668926cc681320b1d2fec191c9e7917384dc4803).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70263224
[Test build #25662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25662/consoleFull) for PR 4070 at commit [`622e45c`](https://github.com/apache/spark/commit/622e45c2bebea7a6206fbf29569e766f57a7f17b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70249152
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25658/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70250150
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25659/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by lianhuiwang <gi...@git.apache.org>.
Github user lianhuiwang commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70489709
@rxin yes, some of etl jobs that has groupby and join operators have been tried to use this feature.most of time that can determine number of partition very well. why jenkins report "java.lang.RuntimeException: spark-core: Binary compatibility check failed!"? can you tell me the reasons of failed? thanks.
@ksakellis I take a look at your works in SPARK-4630 and Can you give some suggestions about this PR? thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70249049
[Test build #25658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25658/consoleFull) for PR 4070 at commit [`fc652a5`](https://github.com/apache/spark/commit/fc652a5aefe4d694d0c3374dce6b3c4415c7be56).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70250923
[Test build #25660 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25660/consoleFull) for PR 4070 at commit [`8b7216f`](https://github.com/apache/spark/commit/8b7216fc539ec656cdcf79b04d34da3b8b708423).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4630][Core]Dynamically determine optima...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4070#issuecomment-70258982
[Test build #25660 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25660/consoleFull) for PR 4070 at commit [`8b7216f`](https://github.com/apache/spark/commit/8b7216fc539ec656cdcf79b04d34da3b8b708423).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HashPartitioner(var partitions: Int) extends Partitioner `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org