You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by drcrallen <gi...@git.apache.org> on 2015/12/16 01:02:23 UTC
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
GitHub user drcrallen opened a pull request:
https://github.com/apache/spark/pull/10319
[SPARK-12330] [CORE] Fix mesos coarse mode cleanup
In the current implementation the mesos coarse scheduler does not wait for the mesos tasks to complete before ending the driver. This causes a race where the task has to finish cleaning up before the mesos driver terminates it with a SIGINT (and SIGKILL after 3 seconds if the SIGINT doesn't work).
This PR causes the mesos coarse scheduler to wait for the mesos tasks to finish (with a timeout defined by `spark.mesos.coarse.shutdown.ms`)
This PR also fixes a regression caused by [SPARK-10987] whereby submitting a shutdown causes a race between the local shutdown procedure and the notification of the scheduler driver disconnection. If the scheduler driver disconnection wins the race, the coarse executor incorrectly exits with status 1 (instead of the proper status 0)
With this patch the mesos coarse scheduler terminates properly, the executors clean up, and the tasks are reported as `FINISHED` in the Mesos console (as opposed to `KILLED` in < 1.6 or `FAILED` in 1.6 and later)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/metamx/spark SPARK-12330
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10319.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10319
----
commit 088a6b585fd5a2a73e9ccc1f3252de4f4e2c5ae9
Author: Charles Allen <ch...@allen-net.com>
Date: 2015-12-15T20:27:45Z
[SPARK-12330] Make CoarseMesosSchedulerBackend wait for executors to cleanup during shutdown
* Adds `spark.mesos.coarse.shutdown.ms` to tune shutdown wait period
commit 5e1b13d0c5a4edcc5fad4db2011debe0585940dd
Author: Charles Allen <ch...@allen-net.com>
Date: 2015-12-15T23:41:02Z
Ignore disconnects notifications during shutdown for CoarseGrainedExecutorBackend
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178162846
@drcrallen This looks great, thanks for fixing the issue. I left a few minor comments. Once you fix them I'll merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-180027619
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165876308
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48013/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178879167
**[Test build #50593 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50593/consoleFull)** for PR 10319 at commit [`75887c4`](https://github.com/apache/spark/commit/75887c4879bbfc7ec251a22f3bccc934fce63a4c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172793031
LGTM! Thanks @drcrallen . @tnachen @andrewor14 please have a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51471696
--- Diff: docs/running-on-mesos.md ---
@@ -387,6 +387,13 @@ See the [configuration page](configuration.html) for information on Spark config
</ul>
</td>
</tr>
+<tr>
+ <td><code>spark.mesos.coarse.shutdown.ms</code></td>
+ <td><code>10000</code> (10 seconds)</td>
+ <td>
+ Time (in ms) to wait for executors to report that they have exited. Setting this too low has the risk of shutting down the Mesos driver (and thereby killing the spark executors) while the executor is still in the process of exiting cleanly.
--- End diff --
actually, I would probably not document this config. I don't really see a use case where users would want to change this. If we decide that users do want to control this (which I don't think they will) then we can always expose it later, but we can't go the other way because of backward compatibility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176948394
**[Test build #50393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50393/consoleFull)** for PR 10319 at commit [`b9e1c77`](https://github.com/apache/spark/commit/b9e1c7753c180147bf8fa628eb63c8516f7ad180).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165203618
So IIUC stop is only invoked when an exception occured or shutdown hook is invoked, where both cases it's an task that didn't really finish and user/system want it to be killed. So I'm not sure if ending in these cases to have TASK_KILLED is an bad idea.
The main reason I'm proposing not to change this is really I don't think sleeping in the stop method with a while is a great idea, as it clearly can block other things going on in the system.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51281443
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,23 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
+ val stopwatch = new Stopwatch()
+ stopwatch.start()
+ // slaveIdsWithExecutors has no memory barrier, so this is eventually consistent
+ while (slaveIdsWithExecutors.nonEmpty &&
+ stopwatch.elapsed(TimeUnit.MILLISECONDS) < shutdownTimeoutMS) {
+ Thread.sleep(100)
+ }
--- End diff --
I thought you used to have a check in the end, that if slaveIdsWithExecutors is non empty we print a warning right? I think that's still valuable to print it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165875292
After more thorough testing, it seems that there is still a race for getting a `FINISHED` vs `KILLED` final status with this patch.
here's an example log where it was reported as killed:
STDERR
```
15/12/18 19:01:34 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
15/12/18 19:01:34 INFO MemoryStore: MemoryStore cleared
15/12/18 19:01:34 INFO BlockManager: BlockManager stopped
15/12/18 19:01:34 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/12/18 19:01:34 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/12/18 19:01:34 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
15/12/18 19:01:34 INFO ShutdownHookManager: Shutdown hook called
15/12/18 19:01:34 INFO ShutdownHookManager: Deleting directory /mesos/slaves/355857ee-1069-480d-baa8-e89427bd5840-S13/frameworks/355857ee-1069-480d-baa8-e89427bd5840-0079/executors/3/runs/b1cb5f55-43f3-4d1d-8ada-18fd29319868/tmp/java/spark-76c38206-57df-46aa-bc2a-4ad143d65959
I1218 19:01:44.844650 94385 exec.cpp:381] Executor asked to shutdown
```
STDOUT
```
2015-12-18T19:01:29.228+0000: Total time for which application threads were stopped: 0.0000720 seconds, Stopping threads took: 0.0000280 seconds
2015-12-18T19:01:34.418+0000: Total time for which application threads were stopped: 0.0202110 seconds, Stopping threads took: 0.0170400 seconds
2015-12-18T19:01:34.895+0000: Total time for which application threads were stopped: 0.0003310 seconds, Stopping threads took: 0.0002190 seconds
2015-12-18T19:01:34.896+0000: Total time for which application threads were stopped: 0.0001710 seconds, Stopping threads took: 0.0000940 seconds
2015-12-18T19:01:34.907+0000: Total time for which application threads were stopped: 0.0002100 seconds, Stopping threads took: 0.0001420 seconds
2015-12-18T19:01:34.907+0000: Total time for which application threads were stopped: 0.0002470 seconds, Stopping threads took: 0.0001740 seconds
2015-12-18T19:01:34.945+0000: Total time for which application threads were stopped: 0.0004670 seconds, Stopping threads took: 0.0000420 seconds
2015-12-18T19:01:34.961+0000: Total time for which application threads were stopped: 0.0001110 seconds, Stopping threads took: 0.0000720 seconds
Heap
PSYoungGen total 39748608K, used 18046635K [0x00007f7749000000, 0x00007f8149000000, 0x00007f8149000000)
eden space 37492224K, 46% used [0x00007f7749000000,0x00007f7b7afe63e8,0x00007f8039580000)
from space 2256384K, 19% used [0x00007f80bf480000,0x00007f80dac44b78,0x00007f8149000000)
to space 2194432K, 0% used [0x00007f8039580000,0x00007f8039580000,0x00007f80bf480000)
ParOldGen total 83886080K, used 70680448K [0x00007f6349000000, 0x00007f7749000000, 0x00007f7749000000)
object space 83886080K, 84% used [0x00007f6349000000,0x00007f7422fe0010,0x00007f7749000000)
PSPermGen total 106496K, used 70405K [0x00007f6339000000, 0x00007f633f800000, 0x00007f6349000000)
object space 106496K, 66% used [0x00007f6339000000,0x00007f633d4c16e8,0x00007f633f800000)
Shutting down
Sending SIGTERM to process tree at pid 94404
Killing the following process trees:
[
-+- 94404 sh -c "/mnt/tmp/spark/bin/spark-class" org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url akka.tcp://sparkDriver@REDACTED/user/CoarseGrainedScheduler --executor-id 355857ee-1069-480d-baa8-e89427bd5840-S13 --hostname REDACTED --cores 13 --app-id 355857ee-1069-480d-baa8-e89427bd5840-0079
\--- 94405 ()
]
Command terminated with signal Terminated (pid: 94404)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178879309
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50593/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165838673
I may have found the root cause of the failure to cleanup blocks. Testing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165809417
@tnachen For some reason the shutdown hooks are not finishing properly if it receives a SIGTERM during shutdown. See logs in https://issues.apache.org/jira/browse/SPARK-12330 where `INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown` is followed by a `ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM`
Fixing that may be a better root cause fix, but I haven't looked into the hadoop shutdown hooks enough to know where the issue could be with that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-179976865
**[Test build #50755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50755/consoleFull)** for PR 10319 at commit [`1181a05`](https://github.com/apache/spark/commit/1181a05a8597f3f8fae7a431b1d661b141d9474c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176848352
I think besides the only comment I have everything else LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-179958972
@drcrallen probably [my comment](https://github.com/apache/spark/pull/10319#discussion_r51693326) and @andrewor14'r reply were buried by the GitHub interface. The consensus is to remove the docs about the new setting. Sorry for the trouble.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176971876
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172698287
Thanks for feedback. I'll get to the fixes here very shortly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176761491
I don't see the point of holding back a review until an import is moved two lines above. Better give the feedback now, so there's not so many back-and-forths. This has been staying in the review queue for long enough, IMO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-179441349
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50667/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-179441344
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-169392267
@tnachen Do you have any suggestions on ways to wait for executors to report as being cleaned up before calling `mesosDriver.stop()`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176338070
This still hasn't run tests yet because it's still failing scala style. Also it looks a little out of date. @drcrallen would you mind rebasing to master and fixing the style violations?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-177964792
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50488/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r47725842
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,22 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (!stopCalled.compareAndSet(false, true)) {
--- End diff --
I don't think we need to make stopCalled atomic boolean since we already have a lock around this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178952067
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178844897
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50592/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176895198
**[Test build #50390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50390/consoleFull)** for PR 10319 at commit [`0a24530`](https://github.com/apache/spark/commit/0a2453085971584cc84e30760d935858c81d7607).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r49696555
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +380,23 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
+ val stopwatch = new Stopwatch()
+ stopwatch.start()
+ // Eventually consistent slaveIdsWithExecutors
--- End diff --
Not sure if this comments says much, can you perhaps add some comments that slaveIdsWithExecutors will be empty once we receive task termination status updates from all tasks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-171409994
also cc @dragos
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165876305
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176895203
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178841453
**[Test build #50593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50593/consoleFull)** for PR 10319 at commit [`75887c4`](https://github.com/apache/spark/commit/75887c4879bbfc7ec251a22f3bccc934fce63a4c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51471308
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,27 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
+ val stopwatch = new Stopwatch()
+ stopwatch.start()
+ // slaveIdsWithExecutors has no memory barrier, so this is eventually consistent
+ while (slaveIdsWithExecutors.nonEmpty &&
+ stopwatch.elapsed(TimeUnit.MILLISECONDS) < shutdownTimeoutMS) {
+ Thread.sleep(100)
+ }
+ if(slaveIdsWithExecutors.nonEmpty) {
+ logWarning(s"${slaveIdsWithExecutors.size} executors still running. "
+ + "Proceeding with mesos driver stop.")
--- End diff --
I don't understand this warning message. I think you mean something more like
```
Timed out on waiting for executors to terminate ($X still running) after $timeout ms.
Proceeding to stop Mesos driver, which may lead to leftover temporary files on the slaves.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51693326
--- Diff: docs/running-on-mesos.md ---
@@ -387,6 +387,13 @@ See the [configuration page](configuration.html) for information on Spark config
</ul>
</td>
</tr>
+<tr>
+ <td><code>spark.mesos.coarse.shutdown.ms</code></td>
+ <td><code>10000</code> (10 seconds)</td>
+ <td>
+ Time (in ms) to wait for executors to report that they have exited. Setting this too low has the risk of shutting down the Mesos driver (and thereby killing the spark executors) while the executor is still in the process of exiting cleanly.
--- End diff --
No, I wasn't really thinking too hard about it. I agree it's not worth making this a user-facing property.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51762702
--- Diff: docs/running-on-mesos.md ---
@@ -387,6 +387,13 @@ See the [configuration page](configuration.html) for information on Spark config
</ul>
</td>
</tr>
+<tr>
+ <td><code>spark.mesos.coarse.shutdown.ms</code></td>
+ <td><code>10000</code> (10 seconds)</td>
+ <td>
+ Time (in ms) to wait for executors to report that they have exited. Setting this too low has the risk of shutting down the Mesos driver (and thereby killing the spark executors) while the executor is still in the process of exiting cleanly.
--- End diff --
@drcrallen can you remove it then?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-173748543
@tnachen / @andrewor14 ping
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-171567791
@drcrallen about waiting suggestions, the best way from the scheduler side is waiting until all tasks are terminated when you like to shutdown. I'm thinking that if this stop is at the very end of the shutdown call, another way is to use a shutdownLock with wait on a timeout here and then just call notify on the TaskStatus update when the map is empty. If the wait exits before it's empty then we know the timeout has hit first. I think we just want to make sure nothing in the scheduler is calling stop by itself, so we don't deadlock.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51930126
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -60,6 +62,12 @@ private[spark] class CoarseMesosSchedulerBackend(
// Maximum number of cores to acquire (TODO: we'll need more flexible controls here)
val maxCores = conf.get("spark.cores.max", Int.MaxValue.toString).toInt
+ private[this] val shutdownTimeoutMS = conf.getTimeAsMs("spark.mesos.coarse.shutdown.ms", "10s")
--- End diff --
by the way I just realized this config is not properly named. Unfortunately I did not catch this during code reviews and I just pushed a hot fix in master to correct this: https://github.com/apache/spark/commit/c756bda477f458ba4aad7fdb2026263507e0ad9b
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r47725778
--- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ---
@@ -45,6 +46,7 @@ private[spark] class CoarseGrainedExecutorBackend(
env: SparkEnv)
extends ThreadSafeRpcEndpoint with ExecutorBackend with Logging {
+ val stopping = new AtomicBoolean(false)
--- End diff --
this can be private
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165889542
The block manager is cleaning up as expected with this patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176874655
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50387/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176444360
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50300/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176948629
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172091910
You're right about the callback for emptying the map, and also about the sleep interval.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165876163
**[Test build #48013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48013/consoleFull)** for PR 10319 at commit [`97de2f1`](https://github.com/apache/spark/commit/97de2f16eb5a8415db9257a538d5d540982a9c8a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-177917664
**[Test build #50488 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50488/consoleFull)** for PR 10319 at commit [`b9e1c77`](https://github.com/apache/spark/commit/b9e1c7753c180147bf8fa628eb63c8516f7ad180).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-177912252
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172504737
Let's try to move this forward. Looks like there's still e few things to do:
- [ ] rebase on master
- [ ] increase the sleep interval
- [ ] document the new setting
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by mgummelt <gi...@git.apache.org>.
Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51927649
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -60,6 +62,12 @@ private[spark] class CoarseMesosSchedulerBackend(
// Maximum number of cores to acquire (TODO: we'll need more flexible controls here)
val maxCores = conf.get("spark.cores.max", Int.MaxValue.toString).toInt
+ private[this] val shutdownTimeoutMS = conf.getTimeAsMs("spark.mesos.coarse.shutdown.ms", "10s")
--- End diff --
Can you please submit a new PR to add this to the docs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178952071
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50616/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176826287
@dragos thanks. My scalastyle keeps failing locally (https://github.com/sbt/sbt/issues/2295), I'll see if I can get it fixed so this stops failing scalastyle.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176972121
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51638713
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,27 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
+ val stopwatch = new Stopwatch()
+ stopwatch.start()
+ // slaveIdsWithExecutors has no memory barrier, so this is eventually consistent
+ while (slaveIdsWithExecutors.nonEmpty &&
+ stopwatch.elapsed(TimeUnit.MILLISECONDS) < shutdownTimeoutMS) {
+ Thread.sleep(100)
+ }
+ if(slaveIdsWithExecutors.nonEmpty) {
+ logWarning(s"${slaveIdsWithExecutors.size} executors still running. "
+ + "Proceeding with mesos driver stop.")
--- End diff --
Modified message. Tried not to use "salves" since that's apparently a bad word in the mesos world
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-164996137
**[Test build #47761 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47761/consoleFull)** for PR 10319 at commit [`5e1b13d`](https://github.com/apache/spark/commit/5e1b13d0c5a4edcc5fad4db2011debe0585940dd).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176895207
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50390/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176892060
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176874651
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r48048840
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -60,6 +63,11 @@ private[spark] class CoarseMesosSchedulerBackend(
// Maximum number of cores to acquire (TODO: we'll need more flexible controls here)
val maxCores = conf.get("spark.cores.max", Int.MaxValue.toString).toInt
+ val shutdownTimeoutMS = conf.getInt("spark.mesos.coarse.shutdown.ms", 10000)
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-164996187
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-164948307
**[Test build #47761 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47761/consoleFull)** for PR 10319 at commit [`5e1b13d`](https://github.com/apache/spark/commit/5e1b13d0c5a4edcc5fad4db2011debe0585940dd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165875644
You can see in the above log entry where the terminal heap information was printed... THEN a SIGTERM was processed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r54175315
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,29 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for executors to report done, or else mesosDriver.stop() will forcefully kill them.
+ // See SPARK-12330
+ val stopwatch = new Stopwatch()
--- End diff --
This Stopwatch constructor was deprecated in newer versions of Guava (https://github.com/google/guava/commit/fd0cbc2c5c90e85fb22c8e86ea19630032090943). In order to work around this issue, I'd like to remove this use of `Stopwatch` since we don't use it anywhere else and it doesn't seem to be buying us a whole lot in the way that it's used here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176443848
**[Test build #50300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50300/consoleFull)** for PR 10319 at commit [`41543e0`](https://github.com/apache/spark/commit/41543e0f67ef7e8a6799857e5f94509f12626e16).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-179968762
@dragos removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172732607
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49659/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176444351
**[Test build #50300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50300/consoleFull)** for PR 10319 at commit [`41543e0`](https://github.com/apache/spark/commit/41543e0f67ef7e8a6799857e5f94509f12626e16).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-171915577
The approach makes sense to me, I will give it a go and report back
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172732603
**[Test build #49659 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49659/consoleFull)** for PR 10319 at commit [`b781297`](https://github.com/apache/spark/commit/b78129727afa11d3b75dd73c8f8384021b0a8239).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176899821
`mvn scalastyle:check` reproduces the error. fixing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-177912182
The error seems spurious:
```
[info] *** 1 TEST FAILED ***
[error] Failed: Total 385, Failed 1, Errors 0, Passed 384, Ignored 2
[error] Failed tests:
[error] org.apache.spark.streaming.InputStreamsSuite
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178951726
**[Test build #50616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50616/consoleFull)** for PR 10319 at commit [`75887c4`](https://github.com/apache/spark/commit/75887c4879bbfc7ec251a22f3bccc934fce63a4c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51470296
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -60,6 +62,12 @@ private[spark] class CoarseMesosSchedulerBackend(
// Maximum number of cores to acquire (TODO: we'll need more flexible controls here)
val maxCores = conf.get("spark.cores.max", Int.MaxValue.toString).toInt
+ private[this] val shutdownTimeoutMS = conf.getInt("spark.mesos.coarse.shutdown.ms", 10000)
--- End diff --
this needs to be
```
conf.getTimeAsMillis("spark.mesos.coarse.shutdownTimeout", "10s")
```
we have a consistent time string format for accepting similar configs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-180027620
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50755/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51287938
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,23 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
+ val stopwatch = new Stopwatch()
+ stopwatch.start()
+ // slaveIdsWithExecutors has no memory barrier, so this is eventually consistent
+ while (slaveIdsWithExecutors.nonEmpty &&
+ stopwatch.elapsed(TimeUnit.MILLISECONDS) < shutdownTimeoutMS) {
+ Thread.sleep(100)
+ }
--- End diff --
added warning if still nonEmpty
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r47812730
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -60,6 +63,11 @@ private[spark] class CoarseMesosSchedulerBackend(
// Maximum number of cores to acquire (TODO: we'll need more flexible controls here)
val maxCores = conf.get("spark.cores.max", Int.MaxValue.toString).toInt
+ val shutdownTimeoutMS = conf.getInt("spark.mesos.coarse.shutdown.ms", 10000)
--- End diff --
private val
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-164945171
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-179981696
LGTM merged into master. The last commit didn't change any code so this can't fail tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178893454
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-179440696
**[Test build #50667 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50667/consoleFull)** for PR 10319 at commit [`75887c4`](https://github.com/apache/spark/commit/75887c4879bbfc7ec251a22f3bccc934fce63a4c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172003851
Regarding the question on waiting strategies, I don't think polling is that bad. It's localized and simple to reason about, and I suppose `stop` should be synchronous. @tnachen's suggestion is fine, but unless I miss something, there's no significant advantage, but there's a potential deadlock situation and more complicated reasoning (`notify` is called from a callback, so looking at `stop` is no longer enough to understand what's going on).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-164945239
@drcrallen mind changing the `[CORE]` in the title to `[MESOS]`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178835670
@andrewor14 Addressed comments except for https://github.com/apache/spark/pull/10319#discussion_r51638882
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r48048809
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,22 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (!stopCalled.compareAndSet(false, true)) {
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-179392112
**[Test build #50667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50667/consoleFull)** for PR 10319 at commit [`75887c4`](https://github.com/apache/spark/commit/75887c4879bbfc7ec251a22f3bccc934fce63a4c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165847162
Nope, SparkEnv.stop() does not block on multiple calls to make sure stop has completed at least once. But even when ensuring that happens the shutdown process still does not end cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r48048812
--- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ---
@@ -45,6 +46,7 @@ private[spark] class CoarseGrainedExecutorBackend(
env: SparkEnv)
extends ThreadSafeRpcEndpoint with ExecutorBackend with Logging {
+ val stopping = new AtomicBoolean(false)
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-177964239
**[Test build #50488 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50488/consoleFull)** for PR 10319 at commit [`b9e1c77`](https://github.com/apache/spark/commit/b9e1c7753c180147bf8fa628eb63c8516f7ad180).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176828137
Not sure if this is what you tried, but you can run only the style checks using `dev/lint-scala`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-179386654
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51638882
--- Diff: docs/running-on-mesos.md ---
@@ -387,6 +387,13 @@ See the [configuration page](configuration.html) for information on Spark config
</ul>
</td>
</tr>
+<tr>
+ <td><code>spark.mesos.coarse.shutdown.ms</code></td>
+ <td><code>10000</code> (10 seconds)</td>
+ <td>
+ Time (in ms) to wait for executors to report that they have exited. Setting this too low has the risk of shutting down the Mesos driver (and thereby killing the spark executors) while the executor is still in the process of exiting cleanly.
--- End diff --
Please reconcile with https://github.com/apache/spark/pull/10319#issuecomment-172006944 on if this needs to be here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172732497
**[Test build #49659 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49659/consoleFull)** for PR 10319 at commit [`b781297`](https://github.com/apache/spark/commit/b78129727afa11d3b75dd73c8f8384021b0a8239).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178844895
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176972208
@drcrallen sorry I don't think everyone has the permissions to trigger jenkins, I'll help watch this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-171356909
@tnachen Ping again regarding question in https://github.com/apache/spark/pull/10319#issuecomment-169392267
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176977605
**[Test build #50410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50410/consoleFull)** for PR 10319 at commit [`b9e1c77`](https://github.com/apache/spark/commit/b9e1c7753c180147bf8fa628eb63c8516f7ad180).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176890203
Failed on git fetch. @dragos how do I fix that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10319
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-171359979
@tnachen I also haven't done a good job at making it more clear previously in this PR that the block manager does not properly cleanup without this patch. See https://issues.apache.org/jira/browse/SPARK-12330 for more info
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172006944
One more thing: Please add the new setting in the Mesos docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51470974
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,27 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
+ val stopwatch = new Stopwatch()
+ stopwatch.start()
+ // slaveIdsWithExecutors has no memory barrier, so this is eventually consistent
+ while (slaveIdsWithExecutors.nonEmpty &&
+ stopwatch.elapsed(TimeUnit.MILLISECONDS) < shutdownTimeoutMS) {
+ Thread.sleep(100)
+ }
+ if(slaveIdsWithExecutors.nonEmpty) {
--- End diff --
style:
```
if (slaveIds...) {
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172731505
@dragos Updated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-164941146
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-180027296
**[Test build #50755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50755/consoleFull)** for PR 10319 at commit [`1181a05`](https://github.com/apache/spark/commit/1181a05a8597f3f8fae7a431b1d661b141d9474c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172041940
I think that either case just looking at stop isn't enough since we are relying on the callback to empty the executors map for us to exit the loop before the timeout, so either way i thought it's just explicit and don't have to busy wait for no reason. I'm fine with a while loop here, I might increase the sleep time since we are waiting on messages to be sent back from executors to driver and I would think checking every 100ms probably enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176526812
@drcrallen sorry for the delay, can you please fix the scala style tests first? will take a look once you update it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178850512
Fail was dumb (couldn't fetch from git). Needs retest
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176903099
**[Test build #50393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50393/consoleFull)** for PR 10319 at commit [`b9e1c77`](https://github.com/apache/spark/commit/b9e1c7753c180147bf8fa628eb63c8516f7ad180).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176948634
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50393/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r54676391
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,29 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for executors to report done, or else mesosDriver.stop() will forcefully kill them.
+ // See SPARK-12330
+ val stopwatch = new Stopwatch()
--- End diff --
Sounds good. Sorry, didn't see this note until now, and it looks like this was already fixed in master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51637351
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,27 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
+ val stopwatch = new Stopwatch()
+ stopwatch.start()
+ // slaveIdsWithExecutors has no memory barrier, so this is eventually consistent
+ while (slaveIdsWithExecutors.nonEmpty &&
+ stopwatch.elapsed(TimeUnit.MILLISECONDS) < shutdownTimeoutMS) {
+ Thread.sleep(100)
+ }
+ if(slaveIdsWithExecutors.nonEmpty) {
--- End diff --
surprised the style catch missed that one. Fixing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176845715
@dragos we can certainly review it first, @andrewor14 has been educating me about the review process and how Spark community typically review things after it clears CI.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178850761
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50593/consoleFull is still running
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-177012559
**[Test build #50410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50410/consoleFull)** for PR 10319 at commit [`b9e1c77`](https://github.com/apache/spark/commit/b9e1c7753c180147bf8fa628eb63c8516f7ad180).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-177964790
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165814882
To clarify... That should solve the block cleanup issue. It will not solve the executor reporting incorrect status.
If the executors are to exit "cleanly" (aka FINNISHED instead of KILLED or FAILED) there needs to be a wait of some kind between spark sending a shutdown and shutting down the mesos driver.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-177012684
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50410/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176869277
```spark charlesallen$ ./dev/lint-scala
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
Scalastyle checks passed.```
so I hope this works
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178897207
**[Test build #50616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50616/consoleFull)** for PR 10319 at commit [`75887c4`](https://github.com/apache/spark/commit/75887c4879bbfc7ec251a22f3bccc934fce63a4c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51287135
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,23 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
+ val stopwatch = new Stopwatch()
+ stopwatch.start()
+ // slaveIdsWithExecutors has no memory barrier, so this is eventually consistent
+ while (slaveIdsWithExecutors.nonEmpty &&
+ stopwatch.elapsed(TimeUnit.MILLISECONDS) < shutdownTimeoutMS) {
+ Thread.sleep(100)
+ }
--- End diff --
sure, will make sure that's in
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-165850037
**[Test build #48013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48013/consoleFull)** for PR 10319 at commit [`97de2f1`](https://github.com/apache/spark/commit/97de2f16eb5a8415db9257a538d5d540982a9c8a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51928513
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -60,6 +62,12 @@ private[spark] class CoarseMesosSchedulerBackend(
// Maximum number of cores to acquire (TODO: we'll need more flexible controls here)
val maxCores = conf.get("spark.cores.max", Int.MaxValue.toString).toInt
+ private[this] val shutdownTimeoutMS = conf.getTimeAsMs("spark.mesos.coarse.shutdown.ms", "10s")
--- End diff --
no https://github.com/apache/spark/pull/10319#discussion_r51693326
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-178879307
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176444359
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-176894846
**[Test build #50390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50390/consoleFull)** for PR 10319 at commit [`0a24530`](https://github.com/apache/spark/commit/0a2453085971584cc84e30760d935858c81d7607).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51470918
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,27 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
--- End diff --
can you expand on this comment? From the code my understanding is that we need to wait until all slaves have properly shutdown before we terminate the Mesos task running the driver. It would be good if this comment could provide more context on why we're doing this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51470346
--- Diff: docs/running-on-mesos.md ---
@@ -387,6 +387,13 @@ See the [configuration page](configuration.html) for information on Spark config
</ul>
</td>
</tr>
+<tr>
+ <td><code>spark.mesos.coarse.shutdown.ms</code></td>
+ <td><code>10000</code> (10 seconds)</td>
+ <td>
+ Time (in ms) to wait for executors to report that they have exited. Setting this too low has the risk of shutting down the Mesos driver (and thereby killing the spark executors) while the executor is still in the process of exiting cleanly.
--- End diff --
documentation needs to be updated accordingly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51638750
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -364,7 +379,27 @@ private[spark] class CoarseMesosSchedulerBackend(
}
override def stop() {
- super.stop()
+ // Make sure we're not launching tasks during shutdown
+ stateLock.synchronized {
+ if (stopCalled) {
+ logWarning("Stop called multiple times, ignoring")
+ return
+ }
+ stopCalled = true
+ super.stop()
+ }
+ // Wait for finish
--- End diff --
Expanded
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51657811
--- Diff: docs/running-on-mesos.md ---
@@ -387,6 +387,13 @@ See the [configuration page](configuration.html) for information on Spark config
</ul>
</td>
</tr>
+<tr>
+ <td><code>spark.mesos.coarse.shutdown.ms</code></td>
+ <td><code>10000</code> (10 seconds)</td>
+ <td>
+ Time (in ms) to wait for executors to report that they have exited. Setting this too low has the risk of shutting down the Mesos driver (and thereby killing the spark executors) while the executor is still in the process of exiting cleanly.
--- End diff --
ah I see. @dragos do you see any potential use case for users to actually set this? I would think the user always wants to wait until his temp files are cleaned up but doesn't want to block forever. The default value of 10s probably satisfies this 99% of the time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [CORE] Fix mesos coarse mode cle...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-164996190
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47761/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-172732605
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by drcrallen <gi...@git.apache.org>.
Github user drcrallen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51910856
--- Diff: docs/running-on-mesos.md ---
@@ -387,6 +387,13 @@ See the [configuration page](configuration.html) for information on Spark config
</ul>
</td>
</tr>
+<tr>
+ <td><code>spark.mesos.coarse.shutdown.ms</code></td>
+ <td><code>10000</code> (10 seconds)</td>
+ <td>
+ Time (in ms) to wait for executors to report that they have exited. Setting this too low has the risk of shutting down the Mesos driver (and thereby killing the spark executors) while the executor is still in the process of exiting cleanly.
--- End diff --
Removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by mgummelt <gi...@git.apache.org>.
Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/10319#discussion_r51929188
--- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala ---
@@ -60,6 +62,12 @@ private[spark] class CoarseMesosSchedulerBackend(
// Maximum number of cores to acquire (TODO: we'll need more flexible controls here)
val maxCores = conf.get("spark.cores.max", Int.MaxValue.toString).toInt
+ private[this] val shutdownTimeoutMS = conf.getTimeAsMs("spark.mesos.coarse.shutdown.ms", "10s")
--- End diff --
ah, nvm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12330] [MESOS] Fix mesos coarse mode cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10319#issuecomment-177012681
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org