You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2012/11/30 19:05:58 UTC
[jira] [Created] (YARN-253) Container launch may fail if no files
were localized
Tom White created YARN-253:
------------------------------
Summary: Container launch may fail if no files were localized
Key: YARN-253
URL: https://issues.apache.org/jira/browse/YARN-253
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.2-alpha
Reporter: Tom White
Assignee: Tom White
This can be demonstrated with DistributedShell. The containers running the shell do not have any files to localize (if there is no shell script to copy) so if they run on a different NM to the AM (which does localize files), then they will fail since the appcache directory does not exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-253) Container launch may fail if no files
were localized
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated YARN-253:
---------------------------
Attachment: YARN-253-test.patch
This change to the DS unit test exposes the problem by changing the mini YARN cluster to have multiple NMs and multiple local directories so that the appcache created by the AM is not necessarily the same one used by the shell container.
Here's the stacktrace showing the failure:
{noformat}
2012-11-30 18:01:02,729 WARN [ContainersLauncher #0] launcher.ContainerLaunch (ContainerLaunch.java:call(247)) - Failed to launch container.
java.io.FileNotFoundException: File /Users/tom/workspace/hadoop-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell/org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell-localDir-nm-2_0/usercache/tom/appcache/application_1354298449311_0001 does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:485)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:996)
at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:150)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:187)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:712)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:708)
at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2361)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:708)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createDir(DefaultContainerExecutor.java:332)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:128)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:242)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:68)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
{noformat}
> Container launch may fail if no files were localized
> ----------------------------------------------------
>
> Key: YARN-253
> URL: https://issues.apache.org/jira/browse/YARN-253
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.2-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: YARN-253-test.patch
>
>
> This can be demonstrated with DistributedShell. The containers running the shell do not have any files to localize (if there is no shell script to copy) so if they run on a different NM to the AM (which does localize files), then they will fail since the appcache directory does not exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-253) Container launch may fail if no files
were localized
Posted by "Roman Shaposhnik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Shaposhnik updated YARN-253:
----------------------------------
Priority: Blocker (was: Major)
> Container launch may fail if no files were localized
> ----------------------------------------------------
>
> Key: YARN-253
> URL: https://issues.apache.org/jira/browse/YARN-253
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.2-alpha
> Reporter: Tom White
> Assignee: Tom White
> Priority: Blocker
> Attachments: YARN-253.patch, YARN-253-test.patch
>
>
> This can be demonstrated with DistributedShell. The containers running the shell do not have any files to localize (if there is no shell script to copy) so if they run on a different NM to the AM (which does localize files), then they will fail since the appcache directory does not exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-253) Container launch may fail if no files
were localized
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507686#comment-13507686 ]
Hadoop QA commented on YARN-253:
--------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12555565/YARN-253.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/202//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/202//console
This message is automatically generated.
> Container launch may fail if no files were localized
> ----------------------------------------------------
>
> Key: YARN-253
> URL: https://issues.apache.org/jira/browse/YARN-253
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.2-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: YARN-253.patch, YARN-253-test.patch
>
>
> This can be demonstrated with DistributedShell. The containers running the shell do not have any files to localize (if there is no shell script to copy) so if they run on a different NM to the AM (which does localize files), then they will fail since the appcache directory does not exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-253) Container launch may fail if no files
were localized
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508685#comment-13508685 ]
Tom White commented on YARN-253:
--------------------------------
I see that you removed the change in MAPREDUCE-4427 because it was an unrelated bug, which made sense in the context of that work. I'm curious to hear if there's a better fix than creating parent directories on demand.
> Container launch may fail if no files were localized
> ----------------------------------------------------
>
> Key: YARN-253
> URL: https://issues.apache.org/jira/browse/YARN-253
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.2-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: YARN-253.patch, YARN-253-test.patch
>
>
> This can be demonstrated with DistributedShell. The containers running the shell do not have any files to localize (if there is no shell script to copy) so if they run on a different NM to the AM (which does localize files), then they will fail since the appcache directory does not exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-253) Container launch may fail if no files
were localized
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507882#comment-13507882 ]
Bikas Saha commented on YARN-253:
---------------------------------
I had posted an identical solution (the code has been refactored since then) in the initial patch for MAPREDUCE-4427. [~vinodkv] pointed out to me that it was a known bug and the fix might be more involved that this simple fix. So I removed that fix from the patch. Would be good to double check with him once. I forget the jira tracking the known issue.
> Container launch may fail if no files were localized
> ----------------------------------------------------
>
> Key: YARN-253
> URL: https://issues.apache.org/jira/browse/YARN-253
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.2-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: YARN-253.patch, YARN-253-test.patch
>
>
> This can be demonstrated with DistributedShell. The containers running the shell do not have any files to localize (if there is no shell script to copy) so if they run on a different NM to the AM (which does localize files), then they will fail since the appcache directory does not exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-253) Container launch may fail if no files
were localized
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated YARN-253:
---------------------------
Attachment: YARN-253.patch
This fixes the problem by creating parent directories if they don't already exist. Without the fix the test would fail about 4 times in 10; with the fix I didn't see a failure.
> Container launch may fail if no files were localized
> ----------------------------------------------------
>
> Key: YARN-253
> URL: https://issues.apache.org/jira/browse/YARN-253
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.2-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: YARN-253.patch, YARN-253-test.patch
>
>
> This can be demonstrated with DistributedShell. The containers running the shell do not have any files to localize (if there is no shell script to copy) so if they run on a different NM to the AM (which does localize files), then they will fail since the appcache directory does not exist.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira