[jira] [Commented] (MAPREDUCE-4490) JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)

["sam liu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13744793#comment-13744793 ] 

sam liu commented on MAPREDUCE-4490:
------------------------------------

Hi,

According to the description, I am trying to provide a patch, as we encountered same issue in our Hadoop cluster. 

First, I added a function in task-controller.c:
int initialize_task(const char* user,
    const char * good_local_dirs, const char *job_id, const char *task_id) {
	// Prepare the attempt directories for the task JVM.
	int result = create_attempt_directories(user, good_local_dirs, job_id, task_id);
	return result;
}

Of cause, I also modified task-controller.h/task-controller.c/main.c. After that, I try to call this feature through ShellCommandExecutor in LinuxTaskController#createLogDir. However, I found the default LinuxTaskController#createLogDir only has two input parameters (TaskAttemptID taskID,boolean isCleanup), and does not satisfy the input parameters of function initialize_task(const char* user,const char * good_local_dirs, const char *job_id, const char *task_id): we can not get user, dir, jobid, taskid from LinuxTaskController#createLogDir.

Any suggestions on the issue which is blocking my progress?

Thanks a lot! 


                
> JVM reuse is incompatible with LinuxTaskController (and therefore incompatible with Security)
> ---------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4490
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4490
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: task-controller, tasktracker
>    Affects Versions: 0.20.205.0, 1.0.3
>            Reporter: George Datskos
>
> When using LinuxTaskController, JVM reuse (mapred.job.reuse.jvm.num.tasks > 1) with more map tasks in a job than there are map slots in the cluster will result in immediate task failures for the second task in each JVM (and then the JVM exits). We have investigated this bug and the root cause is as follows. When using LinuxTaskController, the userlog directory for a task attempt (../userlogs/job/task-attempt) is created only on the first invocation (when the JVM is launched) because userlogs directories are created by the task-controller binary which only runs *once* per JVM. Therefore, attempting to create log.index is guaranteed to fail with ENOENT leading to immediate task failure and child JVM exit.
> {quote}
> 2012-07-24 14:29:11,914 INFO org.apache.hadoop.mapred.TaskLog: Starting logging for a new task attempt_201207241401_0013_m_000027_0 in the same JVM as that of the first task /var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_000006_0
> 2012-07-24 14:29:11,915 WARN org.apache.hadoop.mapred.Child: Error running child
> ENOENT: No such file or directory
>         at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
>         at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
>         at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
>         at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
>         at org.apache.hadoop.mapred.Child.main(Child.java:229)
> {quote}
> The above error occurs in a JVM which runs tasks 6 and 27.  Task6 goes smoothly. Then Task27 starts. The directory /var/log/hadoop/mapred/userlogs/job_201207241401_0013/attempt_201207241401_0013_m_0000027_0 is never created so when mapred.Child tries to write the log.index file for Task27, it fails with ENOENT because the attempt_201207241401_0013_m_0000027_0 directory does not exist. Therefore, the second task in each JVM is guaranteed to fail (and then the JVM exits) every time when using LinuxTaskController. Note that this problem does not occur when using the DefaultTaskController because the userlogs directories are created for each task (not just for each JVM as with LinuxTaskController).
> For each task, the TaskRunner calls the TaskController's createLogDir method before attempting to write out an index file.
> * DefaultTaskController#createLogDir: creates log directory for each task
> * LinuxTaskController#createLogDir: does nothing
> ** task-controller binary creates log directory [create_attempt_directories] (but only for the first task)
> Possible Solution: add a new command to task-controller *initialize task* to create attempt directories.  Call that command, with ShellCommandExecutor, in the LinuxTaskController#createLogDir method

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira