You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by uncleGen <gi...@git.apache.org> on 2017/01/24 10:40:23 UTC
[GitHub] spark pull request #16691: [SPARK-19349][DStreams] Check resource ready to a...
GitHub user uncleGen opened a pull request:
https://github.com/apache/spark/pull/16691
[SPARK-19349][DStreams] Check resource ready to avoid multiple receivers to be scheduled on the same node.
## What changes were proposed in this pull request?
remove related TODO
Currently, we can only ensure registered resource satisfy the "spark.scheduler.minRegisteredResourcesRatio". But if "spark.scheduler.minRegisteredResourcesRatio" is set too small, receivers may still be scheduled to few nodes. In fact, we can give once more chance to wait for sufficient resource to schedule receiver evenly.
## How was this patch tested?
existing ut
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uncleGen/spark SPARK-19349
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16691.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16691
----
commit 0c24291b2738d2c71b59decc60b9e33524b8f84d
Author: uncleGen <hu...@gmail.com>
Date: 2017-01-24T10:37:15Z
SPARK-19349: Check resource ready to avoid multiple receivers to be scheduled on the same node.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16691
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71971/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams]improve resource ready check to a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/16691
I don't think all cases need to wait for all resources ready. That's why `spark.scheduler.minRegisteredResourcesRatio` is added to allow the user changing the behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16691
Working on the test failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16691
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71930/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16691: [SPARK-19349][DStreams]improve resource ready che...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen closed the pull request at:
https://github.com/apache/spark/pull/16691
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16691
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16691
**[Test build #71956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71956/testReport)** for PR 16691 at commit [`0c24291`](https://github.com/apache/spark/commit/0c24291b2738d2c71b59decc60b9e33524b8f84d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams]improve resource ready check to a...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16691
cc @tdas also
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16691
**[Test build #71956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71956/testReport)** for PR 16691 at commit [`0c24291`](https://github.com/apache/spark/commit/0c24291b2738d2c71b59decc60b9e33524b8f84d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16691
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams]improve resource ready check to a...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/16691
I don't like copying codes from Spark core. Maybe let's still keep using a dummy Spark job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16691
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16691: [SPARK-19349][DStreams] Check resource ready to a...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16691#discussion_r97518189
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala ---
@@ -422,16 +423,36 @@ class ReceiverTracker(ssc: StreamingContext, skipReceiverLaunch: Boolean = false
}
/**
- * Run the dummy Spark job to ensure that all slaves have registered. This avoids all the
- * receivers to be scheduled on the same node.
+ * Wait for executors register ready. This avoids multiple receivers to be scheduled
+ * on the same node. Here, we check whether all resource has been registered. If not,
+ * and the number of receiver is larger than the number of registered executors, we
+ * will give once more chance to wait for remaining executors to register for
+ * "spark.scheduler.maxRegisteredResourcesWaitingTime" times.
*
- * TODO Should poll the executor number and wait for executors according to
- * "spark.scheduler.minRegisteredResourcesRatio" and
- * "spark.scheduler.maxRegisteredResourcesWaitingTime" rather than running a dummy job.
+ * This only occurs when set too small "spark.scheduler.minRegisteredResourcesRatio".
*/
- private def runDummySparkJob(): Unit = {
+ private def checkResourceReady(): Unit = {
+ val pollTime = 100
+ val checkingStarted = System.currentTimeMillis()
+ val onceMoreWaitingTimeMs =
+ ssc.sparkContext.conf.getTimeAsMs("spark.scheduler.maxRegisteredResourcesWaitingTime", "30s")
+
--- End diff --
here use "spark.scheduler.maxRegisteredResourcesWaitingTime", need specific config\uff1f
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16691
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16691
**[Test build #71971 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71971/testReport)** for PR 16691 at commit [`f7312c7`](https://github.com/apache/spark/commit/f7312c76902836d0341a9ca0dcb1412ac413f573).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16691
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71956/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16691
**[Test build #71930 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71930/testReport)** for PR 16691 at commit [`0c24291`](https://github.com/apache/spark/commit/0c24291b2738d2c71b59decc60b9e33524b8f84d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16691
**[Test build #71971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71971/testReport)** for PR 16691 at commit [`f7312c7`](https://github.com/apache/spark/commit/f7312c76902836d0341a9ca0dcb1412ac413f573).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16691
**[Test build #71930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71930/testReport)** for PR 16691 at commit [`0c24291`](https://github.com/apache/spark/commit/0c24291b2738d2c71b59decc60b9e33524b8f84d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams]improve resource ready check to a...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16691
@zsxwing That is right, this pr is the improvement for the previous one. There is no much performance regression, i.e. only wait for "spark.scheduler.maxRegisteredResourcesWaitingTime" for resource ready (enough executors for receivers or satisfy the "minRegisteredResourcesRatio")
Any suggestion is appreciated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16691: [SPARK-19349][DStreams] Check resource ready to avoid mu...
Posted by uncleGen <gi...@git.apache.org>.
Github user uncleGen commented on the issue:
https://github.com/apache/spark/pull/16691
cc @zsxwing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org