[jira] [Created] (SPARK-21991) [LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load

["Andrea Zito (JIRA)" <ji...@apache.org>]

Andrea Zito created SPARK-21991:
-----------------------------------

             Summary: [LAUNCHER] LauncherServer acceptConnections thread sometime dies if machine has very high load
                 Key: SPARK-21991
                 URL: https://issues.apache.org/jira/browse/SPARK-21991
             Project: Spark
          Issue Type: Bug
          Components: Spark Submit
    Affects Versions: 2.2.0, 2.1.1, 2.1.0, 2.0.2
         Environment: Single node machine running Ubuntu 16.04.2 LTS (4.4.0-79-generic)
YARN 2.7.2
Spark 2.0.2
            Reporter: Andrea Zito
            Priority: Minor


The way the _LauncherServer_ _acceptConnections_ thread schedules client timeouts causes (non-deterministically) the thread to die with the following exception if the machine is under very high load:

{noformat}
Exception in thread "LauncherServer-1" java.lang.IllegalStateException: Task already scheduled or cancelled
        at java.util.Timer.sched(Timer.java:401)
        at java.util.Timer.schedule(Timer.java:193)
        at org.apache.spark.launcher.LauncherServer.acceptConnections(LauncherServer.java:249)
        at org.apache.spark.launcher.LauncherServer.access$000(LauncherServer.java:80)
        at org.apache.spark.launcher.LauncherServer$1.run(LauncherServer.java:143)
{noformat}

The issue is related to the ordering of actions that the _acceptConnections_ thread uses to handle a client connection:

# create timeout action
# create client thread
# start client thread
# schedule timeout action

Under normal conditions the scheduling of the timeout action happen before the client thread has a chance to start, however if the machine is under very high load the client thread can receive CPU time before the timeout action gets scheduled.

If this condition happen, the client thread cancel the timeout action (which is not yet been scheduled) and goes on, but as soon as the _acceptConnections_ thread gets the CPU back, it will try to schedule the timeout action (which has already been canceled) thus raising the exception.

Changing the order in which the client thread gets started and the timeout gets scheduled seems to be sufficient to fix this issue.

As stated above the issue is non-deterministic, I faced the issue multiple times on a single-node machine submitting a high number of short jobs sequentially, but I couldn't easily create a test reproducing the issue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org