You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by nivox <gi...@git.apache.org> on 2017/09/13 09:16:32 UTC
[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...
GitHub user nivox opened a pull request:
https://github.com/apache/spark/pull/19217
[SPARK-21991][LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load
## What changes were proposed in this pull request?
This patch changes the order in which _acceptConnections_ starts the client thread and schedules the client timeout action ensuring that the latter has been scheduled before the former get a chance to cancel it.
## How was this patch tested?
Due to the non-deterministic nature of the patch I wan't able to add a new test for this issue.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nivox/spark SPARK-21991
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19217.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19217
----
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] Fix race condition in Lau...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19217
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19217
That sounds better.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...
Posted by nivox <gi...@git.apache.org>.
Github user nivox commented on the issue:
https://github.com/apache/spark/pull/19217
@vanzin @ash211 I just modified the title of the PR as per your suggestion
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...
Posted by nivox <gi...@git.apache.org>.
Github user nivox commented on a diff in the pull request:
https://github.com/apache/spark/pull/19217#discussion_r146518809
--- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
@@ -232,20 +232,20 @@ public void run() {
};
ServerConnection clientConnection = new ServerConnection(client, timeout);
Thread clientThread = factory.newThread(clientConnection);
- synchronized (timeout) {
- clientThread.start();
- synchronized (clients) {
- clients.add(clientConnection);
- }
- long timeoutMs = getConnectionTimeout();
- // 0 is used for testing to avoid issues with clock resolution / thread scheduling,
- // and force an immediate timeout.
- if (timeoutMs > 0) {
- timeoutTimer.schedule(timeout, getConnectionTimeout());
- } else {
- timeout.run();
- }
+ synchronized (clients) {
+ clients.add(clientConnection);
--- End diff --
Adding the connection to `clients` *before* starting the thread should be safer since otherwise the thread could have a chance of running and terminating *before* the connection is added to `clients`. This would cause the addition of an already terminated connection to the `clients` list which nobody would ever cleanup (i.e. memory leak).
This situation is really unlikely but still a possibility.
Changing the order of operation shouldn't affect any logic since the `clients` list is only used for cleanup in the `close` method.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the issue:
https://github.com/apache/spark/pull/19217
How about `[SPARK-21991][LAUNCHER] Fix race condition in LauncherServer#acceptConnections` ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the issue:
https://github.com/apache/spark/pull/19217
https://github.com/apache/spark/pull/19574
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19217
**[Test build #81741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81741/testReport)** for PR 19217 at commit [`72af1aa`](https://github.com/apache/spark/commit/72af1aa212b645cd771e1b511f4c428f73743994).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...
Posted by nivox <gi...@git.apache.org>.
Github user nivox commented on a diff in the pull request:
https://github.com/apache/spark/pull/19217#discussion_r146519805
--- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
@@ -232,20 +232,20 @@ public void run() {
};
ServerConnection clientConnection = new ServerConnection(client, timeout);
Thread clientThread = factory.newThread(clientConnection);
- synchronized (timeout) {
--- End diff --
This was exactly my reasoning. I couldn't find any reason why the synchronisation was needed. The removal was only a means to make the code simpler removing the cognitive cost suggested by the presence of a synchronise keyword. The actual fix doesn't depend on it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19217#discussion_r146493105
--- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
@@ -232,20 +232,20 @@ public void run() {
};
ServerConnection clientConnection = new ServerConnection(client, timeout);
Thread clientThread = factory.newThread(clientConnection);
- synchronized (timeout) {
--- End diff --
we're no longer synchronizing on the `timeout` here, but I didn't see anywhere else synchronizing on it either (including `ServerConnection`). Given it doesn't escape this method, I'm not sure how multiple threads could ever access `timeout` at once, so it makes sense to remove this synchronization
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19217
Merged to 2.0 also.
I have a feeling that maven builds might fail because I noticed some trailing whitespace when looking at the raw patch...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19217#discussion_r146492057
--- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
@@ -232,20 +232,20 @@ public void run() {
};
ServerConnection clientConnection = new ServerConnection(client, timeout);
Thread clientThread = factory.newThread(clientConnection);
- synchronized (timeout) {
- clientThread.start();
- synchronized (clients) {
- clients.add(clientConnection);
- }
- long timeoutMs = getConnectionTimeout();
- // 0 is used for testing to avoid issues with clock resolution / thread scheduling,
- // and force an immediate timeout.
- if (timeoutMs > 0) {
- timeoutTimer.schedule(timeout, getConnectionTimeout());
- } else {
- timeout.run();
- }
+ synchronized (clients) {
+ clients.add(clientConnection);
--- End diff --
we are now adding to `clients` before starting the `clientThread` instead of after. What's the expected ordering for these two operations?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19217
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19217
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/19217
Looks reasonable to me. @vanzin was this OK with you?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19217
The code LGTM, but the PR title should describe the fix, not the bug.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19217
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19217
**[Test build #81741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81741/testReport)** for PR 19217 at commit [`72af1aa`](https://github.com/apache/spark/commit/72af1aa212b645cd771e1b511f4c428f73743994).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the issue:
https://github.com/apache/spark/pull/19217
```
========================================================================
Running Java style checks
========================================================================
Using `mvn` from path: /home/ubuntu/spark/build/apache-maven-3.5.0/bin/mvn
Checkstyle checks failed at following occurrences:
[ERROR] src/main/java/org/apache/spark/launcher/LauncherServer.java:[238] (regexp) RegexpSingleline: No trailing whitespace allowed.
[ERROR] src/main/java/org/apache/spark/launcher/LauncherServer.java:[247] (regexp) RegexpSingleline: No trailing whitespace allowed.
[error] running /home/ubuntu/spark/dev/lint-java ; received return code 1
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19217
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81741/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the issue:
https://github.com/apache/spark/pull/19217
@nivox can you please update the PR title when you get the chance?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19217#discussion_r146493161
--- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
@@ -232,20 +232,20 @@ public void run() {
};
ServerConnection clientConnection = new ServerConnection(client, timeout);
Thread clientThread = factory.newThread(clientConnection);
- synchronized (timeout) {
- clientThread.start();
- synchronized (clients) {
- clients.add(clientConnection);
- }
- long timeoutMs = getConnectionTimeout();
- // 0 is used for testing to avoid issues with clock resolution / thread scheduling,
- // and force an immediate timeout.
- if (timeoutMs > 0) {
- timeoutTimer.schedule(timeout, getConnectionTimeout());
- } else {
- timeout.run();
- }
+ synchronized (clients) {
+ clients.add(clientConnection);
+ }
+
+ long timeoutMs = getConnectionTimeout();
+ // 0 is used for testing to avoid issues with clock resolution / thread scheduling,
+ // and force an immediate timeout.
+ if (timeoutMs > 0) {
+ timeoutTimer.schedule(timeout, timeoutMs);
--- End diff --
+1 on not calling `getConnectionTimeout()` multiple times
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19217
Merging to master / 2.2 / 2.1.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org