You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by davies <gi...@git.apache.org> on 2014/07/30 02:15:25 UTC
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/1643
[SPARK-1740] [PySpark] kill the python worker
Kill only the python worker related to cancelled tasks.
The daemon will start a background thread to monitor all the opened sockets for all workers. If the socket is closed by JVM, this thread will kill the worker.
When an task is cancelled, the socket to worker will be closed, then the worker will be killed by deamon.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark kill
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1643.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1643
----
commit 9c62c675e1761532c171aaba4f030aa6b28f311d
Author: Davies Liu <da...@gmail.com>
Date: 2014-07-30T00:07:25Z
kill the python worker
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50976695
QA tests have started for PR 1643. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17787/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-51013376
@JoshRosen Thanks to review this, your comments help me a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50643871
QA tests have started for PR 1643. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17457/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15595048
--- Diff: python/pyspark/daemon.py ---
@@ -50,6 +54,43 @@ def compute_real_exit_code(exit_code):
return 1
+def check_socket(sockets):
--- End diff --
Maybe call this "check_sockets", and add a method comment about what it's doing and what "sockets" is (that being (pid, sock) pairs)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15731069
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -107,7 +113,9 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
// Wait for it to connect to our socket
serverSocket.setSoTimeout(10000)
try {
- return serverSocket.accept()
+ val socket = serverSocket.accept()
+ simpleWorkers.put(socket, pb)
--- End diff --
It looks like `simpleWorkers` is declared as a map of `Process` but here you're storing a `ProcessBuilder`; IntelliJ displays this as an error, but it still seems to compile. Any idea what's going on here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50718956
I will wait for your patch, and think about using PIDs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50700224
I had tried these two approach, we have not easy way to get PIDs of workers, because it's platform dependent. Sending command to Python daemon will also need the identity of worker, it will need another channel to send commands.
If there are exceptions in Java reading/writing threads, which may lead to close the socket, so we still need this kind of method to kill worker if the socket is closed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50686180
QA tests have started for PR 1643. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17493/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15599426
--- Diff: python/pyspark/daemon.py ---
@@ -50,6 +54,43 @@ def compute_real_exit_code(exit_code):
return 1
+def check_socket(sockets):
+ fds = [sock.fileno() for _, sock in sockets]
+ # poll all the sockets check if some of them were closed
+ avail_fds = select.select(fds, [], [], 1)[0]
+
+ for fd in avail_fds:
+ pid, sock = sockets[fds.index(fd)]
+ try:
+ n = sock.recv(1, socket.MSG_PEEK | socket.MSG_DONTWAIT)
--- End diff --
I was wondering if the kernel does anything special to tie sockets to processes (similar to Java lock biasing). I was asking why we bother reading n, why not just set something like `socketClosed = False` which we set to True in the except condition, rather than reusing this variable n.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15731017
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -189,19 +197,34 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
private def stopDaemon() {
synchronized {
- // Request shutdown of existing daemon by sending SIGTERM
- if (daemon != null) {
- daemon.destroy()
- }
+ if (useDaemon) {
+ // Request shutdown of existing daemon by sending SIGTERM
+ if (daemon != null) {
+ daemon.destroy()
+ }
- daemon = null
- daemonPort = 0
+ daemon = null
+ daemonPort = 0
+ } else {
+ simpleWorkers.mapValues(_.destroy())
+ }
}
}
def stop() {
stopDaemon()
}
+
+ def stopWorker(worker: Socket) {
+ if (useDaemon) {
+ daemonWorkers.get(worker).foreach {
--- End diff --
The other accesses of `daemonWorkers` are guarded by `synchronized` blocks; does this access also need synchronization? It looks like calls to `stopWorker()` only occur from `destroyPythonWorker()`, which is synchronized using the SparkEnv object. To be on the safe side, we should probably add `synchronized` here unless there's a good reason not to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50973341
This looks good overall and I'd say it's ready to merge once we address my last two comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15730979
--- Diff: python/pyspark/daemon.py ---
@@ -141,13 +142,17 @@ def handle_sigchld(*args):
os._exit(1)
else:
os._exit(0)
- else:
+ elif pid > 0:
sock.close()
+ else:
+ raise OSError("fork failed")
--- End diff --
I think that `os.fork()` already handles negative return values by throwing `OSError`, so I think this `else` block is dead code: https://docs.python.org/2/library/os.html#os.fork
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50708471
I think that some of the existing `daemon.py` code is unnecessarily complicated and confusing (see some of my comments at https://github.com/mesos/spark/pull/563), so I'd like to clean things up before we add any more code. I'll have a patch for this cleanup by tonight or tomorrow.
One note: I'm a bit wary of the mixture of fork() and Thread in the same daemon's forked process; is this safe?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15731478
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -107,7 +113,9 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
// Wait for it to connect to our socket
serverSocket.setSoTimeout(10000)
try {
- return serverSocket.accept()
+ val socket = serverSocket.accept()
+ simpleWorkers.put(socket, pb)
--- End diff --
Yep, this looks like a compiler bug. This file compiles in 2.10.4 and gives the expected error in 2.11.2:
```scala
import scala.collection.JavaConversions._
import java.lang.Process
import java.lang.ProcessBuilder
object ImplicitBug {
def main(args: Seq[String]) {
val simpleWorkers = new scala.collection.mutable.WeakHashMap[Int, Process]()
val pb = new ProcessBuilder()
simpleWorkers.put(1, pb)
}
}
```
For the curious, here's the actual implicit conversions that led to this bug (run `/scala-2.10.4/bin/scalac -Xprint:typer ImplicitBug.scala` to get this output):
```scala
[[syntax trees at end of typer]] // ImplicitBug.scala
package <empty> {
import scala.collection.JavaConversions._;
import java.lang.Process;
import java.lang.ProcessBuilder;
object ImplicitBug extends scala.AnyRef {
def <init>(): ImplicitBug.type = {
ImplicitBug.super.<init>();
()
};
def main(args: Seq[String]): Unit = {
val simpleWorkers: scala.collection.mutable.WeakHashMap[Int,Process] = new scala.collection.mutable.WeakHashMap[Int,Process]();
val pb: ProcessBuilder = new java.lang.ProcessBuilder();
{
// Somehow it became a map with `Object` values:
scala.collection.JavaConversions.mapAsJavaMap[Int, Object](simpleWorkers).put(1, pb);
()
}
}
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-51005638
Thanks for updating this! In my earlier review, I had overlooked that the kill involved forking the JVM; your new approach of having the daemon kill the workers is much better.
The test case looks good, too (clever use of Python's `for ... else` construct; I hadn't seen that before).
In https://github.com/apache/spark/pull/1680, there was some discussion over whether to use SIGKILL vs. SIGHUP to kill the Python workers. Now that I've had more time to think about it, I think SIGKILL is a fine approach:
- Spark doesn't provide any documented, user-facing mechanisms for allowing tasks to perform cleanup work when they're cancelled.
- The only case where it might make sense to have a cleanup mechanism is when performing side-effects, such as writing to an external database. A machine could immediately lose power or otherwise fail without executing the cleanup mechanism, so users already have to guard against the effects of immediate failures (e.g. by using transactions).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50954392
@JoshRosen I had redo this PR based your cleanup, plz review again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50691032
QA results for PR 1643:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17493/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50572879
QA tests have started for PR 1643. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17413/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15731276
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -107,7 +113,9 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
// Wait for it to connect to our socket
serverSocket.setSoTimeout(10000)
try {
- return serverSocket.accept()
+ val socket = serverSocket.accept()
+ simpleWorkers.put(socket, pb)
--- End diff --
As an experiment, I added the line
```scala
simpleWorkers(socket).destroy()
```
and, as expected, this results in a `java.lang.ProcessBuilder cannot be cast to java.lang.Process` error.
The compiler should have prevented this, so I think we've found a compiler bug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50703014
I don't think we have platform-dependent problems using PIDs, since Windows doesn't use the daemon to launch its PySpark workers; we only have to support Unix platforms in this code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15595383
--- Diff: python/pyspark/daemon.py ---
@@ -50,6 +54,43 @@ def compute_real_exit_code(exit_code):
return 1
+def check_socket(sockets):
+ fds = [sock.fileno() for _, sock in sockets]
+ # poll all the sockets check if some of them were closed
+ avail_fds = select.select(fds, [], [], 1)[0]
+
+ for fd in avail_fds:
+ pid, sock = sockets[fds.index(fd)]
+ try:
+ n = sock.recv(1, socket.MSG_PEEK | socket.MSG_DONTWAIT)
--- End diff --
If an Exception raised, n is not defined.
This threads will sleep 0.1 after every poll, the overhead will be low, it should not effect another process reading the socket.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15602183
--- Diff: python/pyspark/daemon.py ---
@@ -50,6 +54,43 @@ def compute_real_exit_code(exit_code):
return 1
+def check_socket(sockets):
+ fds = [sock.fileno() for _, sock in sockets]
+ # poll all the sockets check if some of them were closed
+ avail_fds = select.select(fds, [], [], 1)[0]
+
+ for fd in avail_fds:
+ pid, sock = sockets[fds.index(fd)]
+ try:
+ n = sock.recv(1, socket.MSG_PEEK | socket.MSG_DONTWAIT)
--- End diff --
In most cases, there is no exception, recv() will return 0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50955448
QA results for PR 1643:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17751/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50718757
Good question, it's dangerous to mix threads and fork(), it may be cause dead lock in child process. But in this case, because of GIL, then fork() happens, monitor thread is blocked or sleeping or polling, they are thread safe, so it will be a problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50976755
I had fixed several bugs and improve the kill approach, it has unit tests now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15594399
--- Diff: python/pyspark/daemon.py ---
@@ -50,6 +54,43 @@ def compute_real_exit_code(exit_code):
return 1
+def check_socket(sockets):
+ fds = [sock.fileno() for _, sock in sockets]
+ # poll all the sockets check if some of them were closed
+ avail_fds = select.select(fds, [], [], 1)[0]
+
+ for fd in avail_fds:
+ pid, sock = sockets[fds.index(fd)]
+ try:
+ n = sock.recv(1, socket.MSG_PEEK | socket.MSG_DONTWAIT)
--- End diff --
Does sock.recv return 0 itself?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15594965
--- Diff: python/pyspark/daemon.py ---
@@ -50,6 +54,43 @@ def compute_real_exit_code(exit_code):
return 1
+def check_socket(sockets):
+ fds = [sock.fileno() for _, sock in sockets]
+ # poll all the sockets check if some of them were closed
+ avail_fds = select.select(fds, [], [], 1)[0]
+
+ for fd in avail_fds:
+ pid, sock = sockets[fds.index(fd)]
+ try:
+ n = sock.recv(1, socket.MSG_PEEK | socket.MSG_DONTWAIT)
--- End diff --
Are there any concerns (performance or otherwise) related to two different processes polling on the same file descriptor?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50954410
QA tests have started for PR 1643. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17751/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-51005954
I've merged this into `master` and `branch-1.1`. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50561370
QA results for PR 1643:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17395/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50575076
QA results for PR 1643:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17413/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50650638
QA results for PR 1643:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17457/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50974176
QA results for PR 1643:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17775/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50558090
QA tests have started for PR 1643. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17395/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50693477
Would it be possible to store the PIDs of the workers inside of PythonWorkerFactory and directly kill the workers via SIGTERM? Or send a command to the Python daemon and have it kill the workers? That seems like it would be less complex than this socket-polling approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50972731
QA tests have started for PR 1643. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17775/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1643#discussion_r15731246
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala ---
@@ -189,19 +197,34 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
private def stopDaemon() {
synchronized {
- // Request shutdown of existing daemon by sending SIGTERM
- if (daemon != null) {
- daemon.destroy()
- }
+ if (useDaemon) {
+ // Request shutdown of existing daemon by sending SIGTERM
+ if (daemon != null) {
+ daemon.destroy()
+ }
- daemon = null
- daemonPort = 0
+ daemon = null
+ daemonPort = 0
+ } else {
+ simpleWorkers.mapValues(_.destroy())
+ }
}
}
def stop() {
stopDaemon()
}
+
+ def stopWorker(worker: Socket) {
+ if (useDaemon) {
+ daemonWorkers.get(worker).foreach {
--- End diff --
Actually, I think the current synchronization is fine: every call of PythonWorkerFactory's public methods is guarded by SparkEnv's lock.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1643#issuecomment-50977847
QA results for PR 1643:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17787/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1740] [PySpark] kill the python worker
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1643
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org