[jira] Updated: (HADOOP-4232) Race condition in JVM reuse when more than one slot becomes free

["Devaraj Das (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-4232:
--------------------------------

    Attachment: 4232.patch

Attaching a patch that addresses the issue and cleans up the JVM management a fair bit.
1) The target JVM is reserved in JvmManager.reapJvm and JvmManager.spawnNewJvm. This makes it very predictable (as to which JVM will run which task) 
2) The above makes the TaskTracker.getTask simple. We just need to look up the mapping created earlier as opposed to iterating over the set of INITIALIZED tasks.
3) Due to the above, the task state, INITIALIZED, is now redundant and has been removed.
4) Tweaked the logic for the JVM sleep when there is nothing to execute a bit. Now the JVM goes to a longer sleep (1500 millisec) if it doesn't get a task to run even after 5 consecutive calls to getTask (at 500 millisec interval each). This strategy helps a lot when there are small tasks and the JVM just misses a task because it isn't initialized yet, and goes to a long sleep..


> Race condition in JVM reuse when more than one slot becomes free
> ----------------------------------------------------------------
>
>                 Key: HADOOP-4232
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4232
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.19.0
>
>         Attachments: 4232.patch
>
>
> A race condition exists where there are two or more slots free and there are two or more tasks waiting to run. As an example, consider a case where there are two free slots and there are two tasks waiting to run. JVM_job1 and JVM_job2 are the two idle jvms in memory. A waiting task, task job1_t1, kills the JVM_job2 and spawns a new one, JVM_1_job1. While JVM_1_job1 is initializing (it is marked busy during initialization), JVM_job1 picks this task up and hence this becomes busy as well. Another waiting task, job3_t1 finds both the JVMs busy and doesn't spawn a new JVM.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.