You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2014/12/02 06:00:41 UTC
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/3548
[SPARK-4498] [WIP] Add driver -> master heartbeat to detect exited applications and fix executor failure detection logic
This is a WIP fix for SPARK-4498; this isn't the final fix that I want to merge in, but I'm submitting this now to get early feedback from Jenkins and reviewers. The main idea here is to add a periodic driver -> master heartbeat that both signals driver liveness and carries information on whether it the driver has received executors, which allows us to implement proper "don't kill an application due to failed executors as long as it has some running executors" logic in the master.
See discussion at https://issues.apache.org/jira/browse/SPARK-4498 for context.
Before merging, this needs more comments and tests. Specifically, I need tests to check that the heartbeat's information actually corresponds to the right notion of application progress / liveness. There's also open questions about heartbeat interval configuration and failure thresholds. I'll edit this description to accurately reflect the PR before I remove the `[WIP]` tag.
/cc @markhamstra @aarondav @andrewor14 @pwendell @airhorns
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark standalone-failure-detector-interface
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3548.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3548
----
commit 87d7960d660b218a9a965fd7d344e2aae0250128
Author: Josh Rosen <jo...@databricks.com>
Date: 2014-12-02T01:08:17Z
Factor application failure detector logic into own class; add tests.
commit 08746eb02ed6e3d114c56ed77a225a1841e3d7ea
Author: Josh Rosen <jo...@databricks.com>
Date: 2014-12-02T04:52:31Z
[SPARK-4498] [WIP] Add driver -> master heartbeat
commit 418af7ea5e78e2d24104f3cf024f412e1c23bdb6
Author: Josh Rosen <jo...@databricks.com>
Date: 2014-12-02T04:55:36Z
Revert debugging comment
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65199081
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24031/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3548#discussion_r21142226
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -761,6 +769,19 @@ private[spark] class Master(
appId
}
+ /** Check for, and remove, and dead applications */
--- End diff --
and remove, dead
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65285340
I'm going to close this for now in favor of Mark's patch (#3550). There are a couple of ideas here that might be useful for future improvements to this code (including factoring out the policy into a separate file for easier testing, which would be important if we added features like timeout-based host blacklisting), but I agree that this PR is more complex than we need for a narrow fix for this bug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65194229
[Test build #24034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24034/consoleFull) for PR 3548 at commit [`6cc7cad`](https://github.com/apache/spark/commit/6cc7caddc14ea9582c93c619932560bae9fdf37e).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65188508
[Test build #24024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24024/consoleFull) for PR 3548 at commit [`418af7e`](https://github.com/apache/spark/commit/418af7ea5e78e2d24104f3cf024f412e1c23bdb6).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class AppClientHeartbeat(appId: String, hasExecutors: Boolean)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65201367
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24034/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3548#discussion_r21142103
--- Diff: core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala ---
@@ -101,6 +101,14 @@ private[deploy] object DeployMessages {
case class MasterChangeAcknowledged(appId: String)
+ /**
+ * Periodic heartbeat from the app client to the master, used to detect driver death and to
+ * detect drivers that have not obtained any usable executors.
+ * @param appId the application id.
+ * @param hasExecutors true if the app client has at least one running executor, false otherwise.
--- End diff --
maybe `hasRegisteredExecutors` to be more specific. We don't actually care if they're running any tasks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3548#discussion_r21142191
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ---
@@ -761,6 +769,19 @@ private[spark] class Master(
appId
}
+ /** Check for, and remove, and dead applications */
+ def timeOutDeadApplications() {
--- End diff --
`: Unit = {`, sorry
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65202792
[Test build #24035 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24035/consoleFull) for PR 3548 at commit [`9dce0d4`](https://github.com/apache/spark/commit/9dce0d4ad5fcb40a227d9a2182439623ca0adfe8).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class AppClientHeartbeat(appId: String, hasRegisteredExecutors: Boolean)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65202804
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24035/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65193703
Seems needlessly complicated to me. I'm still doing tests, but it seems to me that all that is required is https://github.com/apache/spark/pull/3550
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3548#discussion_r21143563
--- Diff: core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala ---
@@ -111,6 +123,20 @@ private[spark] class AppClient(
case x =>
throw new SparkException("Invalid spark URL: " + x)
}
+ import context.dispatcher
+ // Cancel any existing heartbeat timer
+ heartbeatTimer.foreach(_.cancel())
+ // Start a new master heartbeat timer
+ heartbeatTimer = Some(
+ context.system.scheduler.schedule(heartbeatInterval, heartbeatInterval) {
+ if (master != null) { // guard against race condition while switching masters
--- End diff --
Checked the docs and can confirm that we should be sending a self-message: http://doc.akka.io/docs/akka/snapshot/scala/scheduler.html#Some_examples. I'll fix this up now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/3548#discussion_r21142175
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala ---
@@ -43,9 +43,12 @@ private[spark] class ApplicationInfo(
@transient var coresGranted: Int = _
@transient var endTime: Long = _
@transient var appSource: ApplicationSource = _
+ @transient var lastHeartbeat: Long = _
@transient private var nextExecutorId: Int = _
+ val failureDetector = new ApplicationFailureDetector(desc.name, id)
--- End diff --
hm this looks a little out of place. Since this class is fairly isolated, should we just keep a map of it in `Master` instead of cramming it in `ApplicationInfo`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65191401
I'm working on pulling in [SPARK-2424](https://issues.apache.org/jira/browse/SPARK-2424) as well, but I've run into one minor naming snag: what do I call the new threshold? I thought of `maxConsecutiveExecutorFailures` but that name is kind of misleading since it implies that the application will always be terminated if more than that many failures occur, which isn't _always_ the case (an application which reports that it has at least one registered executor will _never_ be terminated by this mechanism). If we want to be _really_ specific, I suppose that `minConsecutiveExecutorFailuresBeforeAppFailure` is better, since it conveys that _at least_ that many failures must occur before we will consider killing the app. That's too long, though. Anyone have a quick suggestion for a better name?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65183893
[Test build #24024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24024/consoleFull) for PR 3548 at commit [`418af7e`](https://github.com/apache/spark/commit/418af7ea5e78e2d24104f3cf024f412e1c23bdb6).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65192639
[Test build #24031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24031/consoleFull) for PR 3548 at commit [`4132892`](https://github.com/apache/spark/commit/413289235942dc5eb7d6020fa795f3db5ecf3c04).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3548#discussion_r21143014
--- Diff: core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala ---
@@ -111,6 +123,20 @@ private[spark] class AppClient(
case x =>
throw new SparkException("Invalid spark URL: " + x)
}
+ import context.dispatcher
+ // Cancel any existing heartbeat timer
+ heartbeatTimer.foreach(_.cancel())
+ // Start a new master heartbeat timer
+ heartbeatTimer = Some(
+ context.system.scheduler.schedule(heartbeatInterval, heartbeatInterval) {
+ if (master != null) { // guard against race condition while switching masters
--- End diff --
Do you think it would be cleaner to have the timer trigger a self-message which triggers the sending of the heartbeat to the driver? This adds more indirection but lets me remove the synchronization / thread-safety stuff.
Akka experts: is that a more idiomatic way of doing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen closed the pull request at:
https://github.com/apache/spark/pull/3548
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65195159
[Test build #24035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24035/consoleFull) for PR 3548 at commit [`9dce0d4`](https://github.com/apache/spark/commit/9dce0d4ad5fcb40a227d9a2182439623ca0adfe8).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3548#discussion_r21142257
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala ---
@@ -43,9 +43,12 @@ private[spark] class ApplicationInfo(
@transient var coresGranted: Int = _
@transient var endTime: Long = _
@transient var appSource: ApplicationSource = _
+ @transient var lastHeartbeat: Long = _
@transient private var nextExecutorId: Int = _
+ val failureDetector = new ApplicationFailureDetector(desc.name, id)
--- End diff --
I considered that, but that's one more map entry to remember to garbage collect when cleaning up applications. Also, the old state for this used to be in ApplicationInfo, so I don't think it's inconsistent to keep the new encapsulation of that state there, too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65183716
I'll be back later tonight to continue working on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65201358
[Test build #24034 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24034/consoleFull) for PR 3548 at commit [`6cc7cad`](https://github.com/apache/spark/commit/6cc7caddc14ea9582c93c619932560bae9fdf37e).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class AppClientHeartbeat(appId: String, hasRegisteredExecutors: Boolean)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65304935
https://github.com/apache/spark/pull/3550 doesn't address SPARK-2424; so if we want to handle that issue in 1.2, then we still need a PR it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498] [WIP] Add driver -> master heartb...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65188514
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24024/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver -> m...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3548#issuecomment-65199074
[Test build #24031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24031/consoleFull) for PR 3548 at commit [`4132892`](https://github.com/apache/spark/commit/413289235942dc5eb7d6020fa795f3db5ecf3c04).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class AppClientHeartbeat(appId: String, hasRegisteredExecutors: Boolean)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org