You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by harishreedharan <gi...@git.apache.org> on 2014/08/18 23:59:56 UTC
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
GitHub user harishreedharan opened a pull request:
https://github.com/apache/spark/pull/2024
[SPARK-3110][YARN] Add a "ha" mode in YARN mode to keep executors in bet...
...ween restarts.
This patch takes an additional argument from the user and tells YARN to restart the AM if it dies and
also keep the same executors around for the restart attempts.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/harishreedharan/spark yarn-restart-am
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2024.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2024
----
commit 03f05a6065af762484e83bf848ef1f5f7217e181
Author: Hari Shreedharan <hs...@apache.org>
Date: 2014-08-18T21:58:01Z
[SPARK-3110][YARN] Add a "ha" mode in YARN mode to keep executors in between restarts.
This patch takes an additional argument from the user and tells YARN to restart the AM if it dies and
also keep the same executors around for the restart attempts.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan closed the pull request at:
https://github.com/apache/spark/pull/2024
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53597412
since there isn't anything to do for this right now, can you close this and just re-open or re-submit a new one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-52997092
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19068/consoleFull) for PR 2024 at commit [`721d316`](https://github.com/apache/spark/commit/721d316e33bde14735f0954ee2ac9d986f940cdb).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53093676
(The pinging part is not yet implemented - that would be the next step). Please see the jira/doc for details:https://issues.apache.org/jira/browse/SPARK-3129
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53297220
@vanzin @tgravescs @tdas If it is fine with you guys (since you are the ones who'd have to review it), I will add the streaming context reconnection part also in this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-52582448
I should have made it clearer. The idea is for long running processes like streaming, you'd want the AM to come back up and reuse the same executors, so it can get the blocks from the memory of the executors because many streaming systems like Flume cannot really replay the data once it has been taken out. Even for others which can, the time period before data "expires" can mean some data could be lost. This is the first step in a series of patches for this one. The next is to get the AM to find the executors. My current plan is to use HDFS to keep track of where the executors are running and then communicate to them via Akka, to get a block list.
I plan to expose this via SparkSubmit as the last step once we have all of the other pieces in place.
You are right, we should add this in Cluster mode too - I will take a look at updating it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-52988897
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19066/consoleFull) for PR 2024 at commit [`1ddd5dc`](https://github.com/apache/spark/commit/1ddd5dc7498dea24731f20351b9935274e14e90d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53455833
Thanks guys! I will just update this one (and its description). Thanks for the feedback!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-52665784
@tgravescs - Sorry, should have done that earlier. I posted a doc here: https://issues.apache.org/jira/browse/SPARK-3129 for more discussion. Once we have agreement, I will create more sub-tasks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53297503
I just fail to see the usefulness of such a small patch without the rest of the work involved being available. The option being added explicitly depends on a feature that is not part of this patch nor part of any other open PR (or, if there is a PR, can you reference it here?).
If by "streaming context reconnection" you mean the code that actually allows running executors to find the new AM, then yes, absolutely, I'd like to see it included in this patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-52631207
Can you perhaps filing the jiras with overall goal and design first if there are more parts to this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53435471
I do like the idea of breaking things up into small patches but I also like the idea of having a good understanding of how the feature will work together.
What happen if someone sets the -ha flag right now? Do the containers stay around forever? I assume at some point they will timeout but until then will be wasting resource.
I don't think this small patch provides much as is so could be combined with more of the functionality. So I would rather see this one closed and this just combined with the next one where we have more context.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-52989298
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19066/consoleFull) for PR 2024 at commit [`1ddd5dc`](https://github.com/apache/spark/commit/1ddd5dc7498dea24731f20351b9935274e14e90d).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53295425
@tgravescs @vanzin Do you have any more comments? If this one looks ok, and can be pulled in - I will start working on the executor reconnect part.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-52571802
Hmmm... since the executors are launched with the driver's address in their command line (ExecutorRunnableUtil::prepareCommand), how will the running executors find the new driver?
(I'm referring to cluster mode, btw, since that's when the driver is running in the AM.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53028142
@vanzin @tgravescs It does look like Client is what runs in YARN cluster mode, so this patch should handle it. Am I missing something?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53296567
@harishreedharan my question is still the same: how do the executors know the new address of the AM? I know you cover that in your document. But how is that addressed in this patch?
It seems that all this is doing is telling Yarn to not kill the executors, just so they can die by themselves since they can't connect to the AM anymore.
I'd rather see something that works end-to-end than just a small patch that introduces a command line option that doesn't do anything useful that I can see at the moment...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53087902
Ah, yes, seems so. Forgot Client.scala has a `main` method.
But in any case, the current patch wouldn't work in cluster mode, because the driver would come up in a different address. (In fact, how would this even work in client mode? Who restarts the driver?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53089849
Yes, it is expected that the driver would come up at a different address. The driver "pings" the executors when it comes up (there is a doc on the jira + some discussion regarding this) - which persist the driver failure. Once the driver pings the executors, it gets the block info stored in executor memory to reconstruct the batch of data which has not been persisted yet.
In client mode, it is tricky because the AM is unmanaged - so YARN will not restart this - so we'd need some kind of watchdog/wrapper - for now, I think Cluster mode is the only one we can think of.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by harishreedharan <gi...@git.apache.org>.
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53297005
It is not. This is the first of a series of patches (which is why we are not exposing the ha parameter as yet). The next patch will actually add the look up in HDFS so the new AM can find the executors, and maybe the authentication and connection aspect. I am splitting this up into smaller patches for ease of review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-52575334
Ah, I see that `Client` doesn't seem to be used in cluster mode, but still, could this be supported in cluster mode? I'd expect people to use cluster mode more often, especially with things like Spark Streaming that are supposed to stay running for a long time.
You'd also probably need a way to expose this flag through SparkSubmit (not sure if that's done automatically but I don't think so), otherwise it would be hard for people to use it. I'd say it's better to have a SparkConf entry for it instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-52995392
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19067/consoleFull) for PR 2024 at commit [`8b09188`](https://github.com/apache/spark/commit/8b0918869fa421bcfff4f98b9cfddbd4590365f7).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53001581
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19067/consoleFull) for PR 2024 at commit [`8b09188`](https://github.com/apache/spark/commit/8b0918869fa421bcfff4f98b9cfddbd4590365f7).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3110][YARN] Add a "ha" mode in YARN mod...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2024#issuecomment-53004612
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19068/consoleFull) for PR 2024 at commit [`721d316`](https://github.com/apache/spark/commit/721d316e33bde14735f0954ee2ac9d986f940cdb).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org