You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by nivox <gi...@git.apache.org> on 2017/09/13 09:16:32 UTC

[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...

GitHub user nivox opened a pull request:

    https://github.com/apache/spark/pull/19217

    [SPARK-21991][LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load

    ## What changes were proposed in this pull request?
    This patch changes the order in which _acceptConnections_ starts the client thread and schedules the client timeout action ensuring that the latter has been scheduled before the former get a chance to cancel it.
    
    ## How was this patch tested?
    Due to the non-deterministic nature of the patch I wan't able to add a new test for this issue.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nivox/spark SPARK-21991

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/19217.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #19217
    
----

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] Fix race condition in Lau...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/19217


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    That sounds better.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...

Posted by nivox <gi...@git.apache.org>.
Github user nivox commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    @vanzin @ash211 I just modified the title of the PR as per your suggestion


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...

Posted by nivox <gi...@git.apache.org>.
Github user nivox commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19217#discussion_r146518809
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
    @@ -232,20 +232,20 @@ public void run() {
             };
             ServerConnection clientConnection = new ServerConnection(client, timeout);
             Thread clientThread = factory.newThread(clientConnection);
    -        synchronized (timeout) {
    -          clientThread.start();
    -          synchronized (clients) {
    -            clients.add(clientConnection);
    -          }
    -          long timeoutMs = getConnectionTimeout();
    -          // 0 is used for testing to avoid issues with clock resolution / thread scheduling,
    -          // and force an immediate timeout.
    -          if (timeoutMs > 0) {
    -            timeoutTimer.schedule(timeout, getConnectionTimeout());
    -          } else {
    -            timeout.run();
    -          }
    +        synchronized (clients) {
    +          clients.add(clientConnection);
    --- End diff --
    
    Adding the connection to `clients` *before* starting the thread should be safer since otherwise the thread could have a chance of running and terminating *before* the connection is added to `clients`. This would cause the addition of an already terminated connection to the `clients` list which nobody would ever cleanup (i.e. memory leak).
    
    This situation is really unlikely but still a possibility.
    
    Changing the order of operation shouldn't affect any logic since the `clients` list is only used for cleanup in the `close` method.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    How about `[SPARK-21991][LAUNCHER] Fix race condition in LauncherServer#acceptConnections` ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...

Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    https://github.com/apache/spark/pull/19574


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    **[Test build #81741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81741/testReport)** for PR 19217 at commit [`72af1aa`](https://github.com/apache/spark/commit/72af1aa212b645cd771e1b511f4c428f73743994).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...

Posted by nivox <gi...@git.apache.org>.
Github user nivox commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19217#discussion_r146519805
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
    @@ -232,20 +232,20 @@ public void run() {
             };
             ServerConnection clientConnection = new ServerConnection(client, timeout);
             Thread clientThread = factory.newThread(clientConnection);
    -        synchronized (timeout) {
    --- End diff --
    
    This was exactly my reasoning. I couldn't find any reason why the synchronisation was needed. The removal was only a means to make the code simpler removing the cognitive cost suggested by the presence of a synchronise keyword. The actual fix doesn't depend on it.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...

Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19217#discussion_r146493105
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
    @@ -232,20 +232,20 @@ public void run() {
             };
             ServerConnection clientConnection = new ServerConnection(client, timeout);
             Thread clientThread = factory.newThread(clientConnection);
    -        synchronized (timeout) {
    --- End diff --
    
    we're no longer synchronizing on the `timeout` here, but I didn't see anywhere else synchronizing on it either (including `ServerConnection`).  Given it doesn't escape this method, I'm not sure how multiple threads could ever access `timeout` at once, so it makes sense to remove this synchronization


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    Merged to 2.0 also.
    
    I have a feeling that maven builds might fail because I noticed some trailing whitespace when looking at the raw patch...


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...

Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19217#discussion_r146492057
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
    @@ -232,20 +232,20 @@ public void run() {
             };
             ServerConnection clientConnection = new ServerConnection(client, timeout);
             Thread clientThread = factory.newThread(clientConnection);
    -        synchronized (timeout) {
    -          clientThread.start();
    -          synchronized (clients) {
    -            clients.add(clientConnection);
    -          }
    -          long timeoutMs = getConnectionTimeout();
    -          // 0 is used for testing to avoid issues with clock resolution / thread scheduling,
    -          // and force an immediate timeout.
    -          if (timeoutMs > 0) {
    -            timeoutTimer.schedule(timeout, getConnectionTimeout());
    -          } else {
    -            timeout.run();
    -          }
    +        synchronized (clients) {
    +          clients.add(clientConnection);
    --- End diff --
    
    we are now adding to `clients` before starting the `clientThread` instead of after.  What's the expected ordering for these two operations?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    Can one of the admins verify this patch?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    Looks reasonable to me. @vanzin was this OK with you?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    The code LGTM, but the PR title should describe the fix, not the bug.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    ok to test


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    **[Test build #81741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81741/testReport)** for PR 19217 at commit [`72af1aa`](https://github.com/apache/spark/commit/72af1aa212b645cd771e1b511f4c428f73743994).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...

Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    ```
    ========================================================================
    Running Java style checks
    ========================================================================
    Using `mvn` from path: /home/ubuntu/spark/build/apache-maven-3.5.0/bin/mvn
    Checkstyle checks failed at following occurrences:
    [ERROR] src/main/java/org/apache/spark/launcher/LauncherServer.java:[238] (regexp) RegexpSingleline: No trailing whitespace allowed.
    [ERROR] src/main/java/org/apache/spark/launcher/LauncherServer.java:[247] (regexp) RegexpSingleline: No trailing whitespace allowed.
    [error] running /home/ubuntu/spark/dev/lint-java ; received return code 1
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81741/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConnections...

Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    @nivox can you please update the PR title when you get the chance?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #19217: [SPARK-21991][LAUNCHER] LauncherServer acceptConn...

Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19217#discussion_r146493161
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java ---
    @@ -232,20 +232,20 @@ public void run() {
             };
             ServerConnection clientConnection = new ServerConnection(client, timeout);
             Thread clientThread = factory.newThread(clientConnection);
    -        synchronized (timeout) {
    -          clientThread.start();
    -          synchronized (clients) {
    -            clients.add(clientConnection);
    -          }
    -          long timeoutMs = getConnectionTimeout();
    -          // 0 is used for testing to avoid issues with clock resolution / thread scheduling,
    -          // and force an immediate timeout.
    -          if (timeoutMs > 0) {
    -            timeoutTimer.schedule(timeout, getConnectionTimeout());
    -          } else {
    -            timeout.run();
    -          }
    +        synchronized (clients) {
    +          clients.add(clientConnection);
    +        }
    +        
    +        long timeoutMs = getConnectionTimeout();
    +        // 0 is used for testing to avoid issues with clock resolution / thread scheduling,
    +        // and force an immediate timeout.
    +        if (timeoutMs > 0) {
    +          timeoutTimer.schedule(timeout, timeoutMs);
    --- End diff --
    
    +1 on not calling `getConnectionTimeout()` multiple times


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:

    https://github.com/apache/spark/pull/19217
  
    Merging to master / 2.2 / 2.1.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org