You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by WangTaoTheTonic <gi...@git.apache.org> on 2015/04/23 11:15:03 UTC
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/5657
[SPARK-7086][Deploy]Do not retry when public service start on port
https://issues.apache.org/jira/browse/SPARK-7086
I just fix in master side and maybe there are more to fix?
//cc @andrewor14
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/WangTaoTheTonic/spark SPARK-7086
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5657.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5657
----
commit a9dbda83019331ebc6a09786f00f94a5b965da50
Author: WangTaoTheTonic <wa...@huawei.com>
Date: 2015-04-23T08:53:22Z
do not retry when service starts on public port
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-96377454
Considering one condition: user submit apps to master with a port config, let's say `spark://somehost:7077`, and let workers connect to master same way. Once master open another port, for instance 7078, it will be unavailable to submit apps or accept works' registeration.
I know that retrying policy decrease probability of failing when launching master but in the meantime it increase the chance for others to connect it.
Besides I have taken a look at start-all.sh, and it pass one specify port to slave node to launch worker. Obviously worker could not take to master when master take a different port with the passed one.
@srowen I thought it must make some trouble if we take that retries on "public" port unless we found other way to solve it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-95578868
[Test build #30835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30835/consoleFull) for PR 5657 at commit [`a9dbda8`](https://github.com/apache/spark/commit/a9dbda83019331ebc6a09786f00f94a5b965da50).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class NumericType extends NativeType `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-95529805
[Test build #30828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30828/consoleFull) for PR 5657 at commit [`a9dbda8`](https://github.com/apache/spark/commit/a9dbda83019331ebc6a09786f00f94a5b965da50).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic closed the pull request at:
https://github.com/apache/spark/pull/5657
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-95505178
[Test build #30828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30828/consoleFull) for PR 5657 at commit [`a9dbda8`](https://github.com/apache/spark/commit/a9dbda83019331ebc6a09786f00f94a5b965da50).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-95554412
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-96194910
This is related to https://github.com/apache/spark/pull/5575. I am not sure this is something we must force on users. There could be decent reasons to retry binding, and it doesn't 'hurt' except to waste a few tries. This is also a little arbitrary to change the behavior on just a few services. I don't think we should do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-95529836
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30828/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-96471737
After taking a look at https://github.com/apache/spark/pull/3314 and discussion with @scwf offline, we both think the "speifty port range for each" idea is better for issue [SPARK-7086](https://issues.apache.org/jira/browse/SPARK-7086) and [SPARK-4449](https://issues.apache.org/jira/browse/SPARK-4449).
So I will close this and keep track at https://github.com/apache/spark/pull/3314.
@srowen Thanks for your comments and nice idea. :smiley:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-96457581
Ah, we don't have this change committed yet: https://github.com/apache/spark/pull/3314 (Or, a variant on this.) The right-er way to fix this is to be able to express a range of ports, which might only include 1 port, in which case there would be no more retries anyway. I suggesting focusing on resolving SPARK-4449 as a way to fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-95578886
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30835/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-95556071
[Test build #30835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30835/consoleFull) for PR 5657 at commit [`a9dbda8`](https://github.com/apache/spark/commit/a9dbda83019331ebc6a09786f00f94a5b965da50).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-96455354
If retry, then master will use another port. We can see it from `Utils.scala`:
>for (offset <- 0 to maxRetries) {
...
((startPort + offset - 1024) % (65536 - 1024)) + 1024
>logWarning(s"Service$serviceString could not bind on port $tryPort. " +
s"Attempting port ${tryPort + 1}.")
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-7086][Deploy]Do not retry when public s...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5657#issuecomment-96454575
This change merely causes it to never retry. It doesn't cause the master to use another port, right? That would be bad for the reason you give, but this is changing the retry property.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org