You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "zhaoyunjiong (JIRA)" <ji...@apache.org> on 2012/12/04 14:40:59 UTC
[jira] [Created] (MAPREDUCE-4843) When using DefaultTaskController,
JobLocalizer not thread safe
zhaoyunjiong created MAPREDUCE-4843:
---------------------------------------
Summary: When using DefaultTaskController, JobLocalizer not thread safe
Key: MAPREDUCE-4843
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: tasktracker
Affects Versions: 1.1.1
Reporter: zhaoyunjiong
Priority: Critical
In our cluster, some times job will failed due to below exception:
Error initializing attempt_201210181806_18566_r_000376_0: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201210181806_18566/job.xml in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
The root cause is JobLocalizer is not thread safe.
In DefaultTaskController.initializeJob method:
JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
but in JobLocalizer, it just simply keep the reference of the conf.
When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but one conf instance.
So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
It will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4843) When using
DefaultTaskController, JobLocalizer not thread safe
Posted by "zhaoyunjiong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510228#comment-13510228 ]
zhaoyunjiong commented on MAPREDUCE-4843:
-----------------------------------------
Above patch is not working. I'm working on new patch.
> When using DefaultTaskController, JobLocalizer not thread safe
> --------------------------------------------------------------
>
> Key: MAPREDUCE-4843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.1
> Reporter: zhaoyunjiong
> Priority: Critical
>
> In our cluster, some times job will failed due to below exception:
> 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)
> The root cause is JobLocalizer is not thread safe.
> In DefaultTaskController.initializeJob method:
> JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
> but in JobLocalizer, it just simply keep the reference of the conf.
> When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
> So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
> Then it will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4843) When using DefaultTaskController,
JobLocalizer not thread safe
Posted by "zhaoyunjiong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhaoyunjiong updated MAPREDUCE-4843:
------------------------------------
Status: Patch Available (was: Open)
Testing patch
> When using DefaultTaskController, JobLocalizer not thread safe
> --------------------------------------------------------------
>
> Key: MAPREDUCE-4843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.1
> Reporter: zhaoyunjiong
> Priority: Critical
> Attachments: MAPREDUCE-4843-branch-1.1.patch
>
>
> In our cluster, some times job will failed due to below exception:
> 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)
> The root cause is JobLocalizer is not thread safe.
> In DefaultTaskController.initializeJob method:
> JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
> but in JobLocalizer, it just simply keep the reference of the conf.
> When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
> So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
> Then it will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4843) When using
DefaultTaskController, JobLocalizer not thread safe
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511029#comment-13511029 ]
Karthik Kambatla commented on MAPREDUCE-4843:
---------------------------------------------
My bad - read the branch name wrong. I applied the patch locally, and verified that the tests that directly use {{DefaultTaskController}} pass - TestTaskTrackerLocalization, TestJvmManager, TestTaskEnvironment.
+1
> When using DefaultTaskController, JobLocalizer not thread safe
> --------------------------------------------------------------
>
> Key: MAPREDUCE-4843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.1
> Reporter: zhaoyunjiong
> Priority: Critical
> Attachments: MAPREDUCE-4843-branch-1.1.patch
>
>
> In our cluster, some times job will failed due to below exception:
> 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)
> The root cause is JobLocalizer is not thread safe.
> In DefaultTaskController.initializeJob method:
> JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
> but in JobLocalizer, it just simply keep the reference of the conf.
> When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
> So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
> Then it will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4843) When using
DefaultTaskController, JobLocalizer not thread safe
Posted by "zhaoyunjiong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509730#comment-13509730 ]
zhaoyunjiong commented on MAPREDUCE-4843:
-----------------------------------------
The fix is very simple:
diff --git a/src/mapred/org/apache/hadoop/mapred/JobLocalizer.java b/src/mapred/org/apache/hadoop/mapred/JobLocalizer.java
index 0802b03..625face 100644
--- a/src/mapred/org/apache/hadoop/mapred/JobLocalizer.java
+++ b/src/mapred/org/apache/hadoop/mapred/JobLocalizer.java
@@ -108,7 +108,7 @@ public class JobLocalizer {
throw new IOException("Cannot initialize for null jobid");
}
this.jobid = jobid;
- this.ttConf = ttConf;
+ this.ttConf = new JobConf(ttConf);
lfs = FileSystem.getLocal(ttConf).getRaw();
this.localDirs = createPaths(user, localDirs);
ttConf.setStrings(JOB_LOCAL_CTXT, localDirs);
> When using DefaultTaskController, JobLocalizer not thread safe
> --------------------------------------------------------------
>
> Key: MAPREDUCE-4843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.1
> Reporter: zhaoyunjiong
> Priority: Critical
>
> In our cluster, some times job will failed due to below exception:
> Error initializing attempt_201210181806_18566_r_000376_0: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201210181806_18566/job.xml in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
> The root cause is JobLocalizer is not thread safe.
> In DefaultTaskController.initializeJob method:
> JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
> but in JobLocalizer, it just simply keep the reference of the conf.
> When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but one conf instance.
> So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
> It will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4843) When using
DefaultTaskController, JobLocalizer not thread safe
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510899#comment-13510899 ]
Hadoop QA commented on MAPREDUCE-4843:
--------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12556081/MAPREDUCE-4843-branch-1.1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3095//console
This message is automatically generated.
> When using DefaultTaskController, JobLocalizer not thread safe
> --------------------------------------------------------------
>
> Key: MAPREDUCE-4843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.1
> Reporter: zhaoyunjiong
> Priority: Critical
> Attachments: MAPREDUCE-4843-branch-1.1.patch
>
>
> In our cluster, some times job will failed due to below exception:
> 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)
> The root cause is JobLocalizer is not thread safe.
> In DefaultTaskController.initializeJob method:
> JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
> but in JobLocalizer, it just simply keep the reference of the conf.
> When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
> So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
> Then it will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4843) When using DefaultTaskController,
JobLocalizer not thread safe
Posted by "zhaoyunjiong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhaoyunjiong updated MAPREDUCE-4843:
------------------------------------
Description:
In our cluster, some times job will failed due to below exception:
2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)
The root cause is JobLocalizer is not thread safe.
In DefaultTaskController.initializeJob method:
JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
but in JobLocalizer, it just simply keep the reference of the conf.
When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
Then it will cause the previous job's job.xml stored at another user's dir.
was:
In our cluster, some times job will failed due to below exception:
Error initializing attempt_201210181806_18566_r_000376_0: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201210181806_18566/job.xml in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
The root cause is JobLocalizer is not thread safe.
In DefaultTaskController.initializeJob method:
JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
but in JobLocalizer, it just simply keep the reference of the conf.
When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but one conf instance.
So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
It will cause the previous job's job.xml stored at another user's dir.
> When using DefaultTaskController, JobLocalizer not thread safe
> --------------------------------------------------------------
>
> Key: MAPREDUCE-4843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.1
> Reporter: zhaoyunjiong
> Priority: Critical
>
> In our cluster, some times job will failed due to below exception:
> 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)
> The root cause is JobLocalizer is not thread safe.
> In DefaultTaskController.initializeJob method:
> JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
> but in JobLocalizer, it just simply keep the reference of the conf.
> When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
> So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
> Then it will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4843) When using DefaultTaskController,
JobLocalizer not thread safe
Posted by "zhaoyunjiong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhaoyunjiong updated MAPREDUCE-4843:
------------------------------------
Attachment: MAPREDUCE-4843-branch-1.1.patch
Update patch.
> When using DefaultTaskController, JobLocalizer not thread safe
> --------------------------------------------------------------
>
> Key: MAPREDUCE-4843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.1
> Reporter: zhaoyunjiong
> Priority: Critical
> Attachments: MAPREDUCE-4843-branch-1.1.patch
>
>
> In our cluster, some times job will failed due to below exception:
> 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)
> The root cause is JobLocalizer is not thread safe.
> In DefaultTaskController.initializeJob method:
> JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
> but in JobLocalizer, it just simply keep the reference of the conf.
> When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
> So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
> Then it will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4843) When using
DefaultTaskController, JobLocalizer not thread safe
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511003#comment-13511003 ]
Karthik Kambatla commented on MAPREDUCE-4843:
---------------------------------------------
[~zhaoyunjiong] The patch looks good. Can you post a patch against trunk for QA to be able to apply it. Also, I was wondering if it would be possible to add a test?
> When using DefaultTaskController, JobLocalizer not thread safe
> --------------------------------------------------------------
>
> Key: MAPREDUCE-4843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.1
> Reporter: zhaoyunjiong
> Priority: Critical
> Attachments: MAPREDUCE-4843-branch-1.1.patch
>
>
> In our cluster, some times job will failed due to below exception:
> 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)
> The root cause is JobLocalizer is not thread safe.
> In DefaultTaskController.initializeJob method:
> JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
> but in JobLocalizer, it just simply keep the reference of the conf.
> When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
> So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
> Then it will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4843) When using
DefaultTaskController, JobLocalizer not thread safe
Posted by "zhaoyunjiong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511019#comment-13511019 ]
zhaoyunjiong commented on MAPREDUCE-4843:
-----------------------------------------
No need for trunk. In hadoop 2.0, the problem doesn't exist.
It's very difficult to test a thread safe problem, even it's not thread safe, in most case it will pass it.
> When using DefaultTaskController, JobLocalizer not thread safe
> --------------------------------------------------------------
>
> Key: MAPREDUCE-4843
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4843
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 1.1.1
> Reporter: zhaoyunjiong
> Priority: Critical
> Attachments: MAPREDUCE-4843-branch-1.1.patch
>
>
> In our cluster, some times job will failed due to below exception:
> 2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)
> The root cause is JobLocalizer is not thread safe.
> In DefaultTaskController.initializeJob method:
> JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
> but in JobLocalizer, it just simply keep the reference of the conf.
> When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
> So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
> Then it will cause the previous job's job.xml stored at another user's dir.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira