You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mccheah <gi...@git.apache.org> on 2015/02/09 22:55:05 UTC
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
GitHub user mccheah opened a pull request:
https://github.com/apache/spark/pull/4481
[SPARK-5697] Configurable registration retry interval and max attempts
Before, the Spark Driver would attempt to connect 3 times to the master,
with 20 seconds between each attempt, and if the master did not respond,
the driver would give up.
In practice, however, users may have a long queue of jobs or a busy
network that makes giving up this early unreasonable. Users should
choose to allow the driver to wait a longer time to eventually have its
registration be processed by the master node. The user does not need to
manually resubmit jobs as often if they set the timeout and number of
retries to higher numbers, as long as they are willing to accept that a
job may fail later if the master truly does crash or become unreachable.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mccheah/spark allow-different-register-timeouts
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4481.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4481
----
commit 4cb5bf8a7f9bd3a2f412819542490e3b91d6aeac
Author: mcheah <mc...@palantir.com>
Date: 2015-02-09T21:52:05Z
[SPARK-5697] Configurable registration retry interval and max attempts
Before, the Spark Driver would attempt to connect 3 times to the master,
with 20 seconds between each attempt, and if the master did not respond,
the driver would give up.
In practice, however, users may have a long queue of jobs or a busy
network that makes giving up this early unreasonable. Users should
choose to allow the driver to wait a longer time to eventually have its
registration be processed by the master node. The user does not need to
manually resubmit jobs as often if they set the timeout and number of
retries to higher numbers, as long as they are willing to accept that a
job may fail later if the master truly does crash or become unreachable.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-73611362
[Test build #27126 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27126/consoleFull) for PR 4481 at commit [`4cb5bf8`](https://github.com/apache/spark/commit/4cb5bf8a7f9bd3a2f412819542490e3b91d6aeac).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah closed the pull request at:
https://github.com/apache/spark/pull/4481
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-73598644
Let me know if I should document these configurations somewhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-93371064
If there's not going to be follow up along these lines, would you mind closing the PR?
(Aside: we now have a new syntax for specifying time properties, FWIW.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-73664422
Messages being lost because of an unreliable network can make the driver's messages hey lost and thus the driver retires even when the master is alive. Also we have a decent number of Spark contexts being created and stopped frequently, which can make the master's akka message queue become a bottleneck.
We have had a few cases when the driver gave up but the master launched executors anyways, meaning if the driver had just waited a little longer, it could have proceeded with the job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-73659165
Could you describe in more detail the circumstances where you see that the master is overloaded and unable to accept connections? In general the master is a very lightweight component in Spark, I'm really surprised there was ever a case where this didn't just succeed the first time (outside of an all-out failure of the master).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-74346577
@mccheah for our use case what values would you set these to? Just bump the retry count up to like 10?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-73611374
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27126/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-74588769
Hey Matt, Sorry I'm still a bit confused. Basically my concern is that we're trying to ignore an underlying bug by adding a configuration option, which is something we really try not to do.
Is it the case that you regularly have unreliable network connectivity inside of your cluster? Spark overall assumes that nodes can establish reliable TCP connections to one another. Do you actually have TCP flows that are terminated from within the network as a regular occurrence? It's very hard for me to imagine a modern hardware cluster where this is the case.
The second explanation you gave was the Akka message queue. Akka in general should be able to process thousands of messages per second which is _way_ more than anyone would reasonably submit to the standalone cluster manager. It's possible that we are in some way blocking inside of our actors in a way that is severely limiting throughput. If that is the case, then we should identify and fix the bug.
Are you seeing specific akka timeouts or some type of error message that could help pin down what is happening? My guess is that there is just something buggy about job submission, and ideally we should fix that instead of trying to add more knobs to turn to work around it.
If you have a reproduction of this behavior that would actually be the best. I.e. a stress test or something that could identify what is going on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-73664585
(typos because I typed this on my phone, will fix in a bit)
(hey = get, retire = retries)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-74439444
Just increase it depending on the environment. Of course the user should exercise discretion - if the retry count needs to be very high there's likely something very wrong in their system.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5697] Configurable registration retry i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4481#issuecomment-73599106
[Test build #27126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27126/consoleFull) for PR 4481 at commit [`4cb5bf8`](https://github.com/apache/spark/commit/4cb5bf8a7f9bd3a2f412819542490e3b91d6aeac).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org