You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by raghavendhra rahul <ra...@gmail.com> on 2012/01/06 12:45:09 UTC
Container Launching
Hi all,
I am trying to write an application master.Is there a way to specify
node1: 10 conatiners
node2: 10 containers
Can we specify this kind of list using the application master????
Also i set request.setHostName("client"); where client is the hostname of a
node
I checked the log to find the following error
java.io.FileNotFoundException: File
/local1/yarn/.yarn/local/usercache/rahul_2011/appcache/
application_1325760852770_0001 does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
at
org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:815)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:700)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:697)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:697)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:122)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:237)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:67)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
i.e containers are only launching within the host where the application
master runs while the other nodes always remain free.
Any ideas.....
Re: Container Launching
Posted by Bing Jiang <ji...@gmail.com>.
Thanks, Vinod. I have submitted to Hadoop-MapReduce Issue.
https://issues.apache.org/jira/browse/MAPREDUCE-3655
在 2012年1月11日 上午7:58，Vinod Kumar Vavilapalli <vi...@hortonworks.com>写道：
> This seems like a code bug, following your conversation.
>
> Raghavendra/Bing, can one of you please create a ticket at
> https://issues.apache.org/jira/secure/CreateIssue!default.jspa under
> Hadoop MapReduce? That way it can get fixed in the next release.
>
> Thanks,
> +Vinod
>
>
> >>> Also i set request.setHostName("client"); where client is the hostname
> of
> >>> a node
> >>> I checked the log to find the following error
> >>> java.io.FileNotFoundException: File
> >>> /local1/yarn/.yarn/local/usercache/rahul_2011/appcache/
> >>> application_1325760852770_0001 does not exist
> >>> at
> >>>
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
> >>> at
> >>> org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:815)
> >>> at
> >>>
> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
> >>> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
> >>> at
> org.apache.hadoop.fs.FileContext$4.next(FileContext.java:700)
> >>> at
> org.apache.hadoop.fs.FileContext$4.next(FileContext.java:697)
> >>> at
> >>>
> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
> >>> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:697)
> >>> at
> >>>
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:122)
> >>> at
> >>>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:237)
> >>> at
> >>>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:67)
> >>> at
> >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >>> at
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >>> at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >>> at java.lang.Thread.run(Thread.java:636)
> >>>
> >>> i.e containers are only launching within the host where the application
> >>> master runs while the other nodes always remain free.
> >>> Any ideas.....
> >>
> >>
> >
> >
> >
> > --
> > Bing Jiang
> > Tel：(86)134-2619-1361
> > National Research Center for Intelligent Computing Systems
> > Institute of Computing technology
> > Graduate University of Chinese Academy of Science
> >
>
--
Bing Jiang
Tel：(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: http://blog.sina.com.cn/jiangbinglover
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
Re: Container Launching
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
This seems like a code bug, following your conversation.
Raghavendra/Bing, can one of you please create a ticket at
https://issues.apache.org/jira/secure/CreateIssue!default.jspa under
Hadoop MapReduce? That way it can get fixed in the next release.
Thanks,
+Vinod
>>> Also i set request.setHostName("client"); where client is the hostname of
>>> a node
>>> I checked the log to find the following error
>>> java.io.FileNotFoundException: File
>>> /local1/yarn/.yarn/local/usercache/rahul_2011/appcache/
>>> application_1325760852770_0001 does not exist
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
>>> at
>>> org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:815)
>>> at
>>> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
>>> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
>>> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:700)
>>> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:697)
>>> at
>>> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
>>> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:697)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:122)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:237)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:67)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> at java.lang.Thread.run(Thread.java:636)
>>>
>>> i.e containers are only launching within the host where the application
>>> master runs while the other nodes always remain free.
>>> Any ideas.....
>>
>>
>
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
Re: Container Launching
Posted by raghavendhra rahul <ra...@gmail.com>.
Thanks that has been solved.
Now the problem is
Is there a way to specify
node1: 10 conatiners
node2: 10 containers
Can we specify this kind of list using the application master????
How to specify setHostName("client")
Is there specific format.
2012/1/9 Bing Jiang <ji...@gmail.com>
> Hi, I have encountered a same question. Look below, and hope it can help u.
> I get help from Hadoop-Yarn to deploy my real-time distributed computation
> system, and I get reply from mapreduce-user@hadoop.apache.org to follow
> these guilders below:
>
>
>>> http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html
>>>
>>> http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
>>>
>> When I follow the steps to construct my Client, ApplicationMaster.And an
> issue occurs to me that NM fail to launch a Container because of
> java.io.FileNotFoundException.
> So the part of NM log has been attached below:
> ....
> 2011-12-29 15:49:16,250 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.application.Application: Adding
> container_1325062142731_0006_01_000001 to application
> application_1325062142731_0006
> 2011-12-29 15:49:16,250 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ApplicationLocalizationEvent.EventType:
> INIT_APPLICATION_RESOURCES
> 2011-12-29 15:49:16,250 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationInitedEvent.EventType:
> APPLICATION_INITED
> 2011-12-29 15:49:16,250 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Processing application_1325062142731_0006 of type APPLICATION_INITED
> 2011-12-29 15:49:16,250 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> Application application_1325062142731_0006 transitioned from INITING to
> RUNNING
> 2011-12-29 15:49:16,250 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerAppStartedEvent.EventType:
> APPLICATION_STARTED
> 2011-12-29 15:49:16,250 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerInitEvent.EventType:
> INIT_CONTAINER
> 2011-12-29 15:49:16,250 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Processing container_1325062142731_0006_01_000001 of type INIT_CONTAINER
> 2011-12-29 15:49:16,250 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1325062142731_0006_01_000001 transitioned from NEW to
> LOCALIZED
> 2011-12-29 15:49:16,250 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEvent.EventType:
> LAUNCH_CONTAINER
> 2011-12-29 15:49:16,287 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEvent.EventType:
> CONTAINER_LAUNCHED
> 2011-12-29 15:49:16,287 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Processing container_1325062142731_0006_01_000001 of type CONTAINER_LAUNCHED
> 2011-12-29 15:49:16,287 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1325062142731_0006_01_000001 transitioned from
> LOCALIZED to RUNNING
> 2011-12-29 15:49:16,288 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerStartMonitoringEvent.EventType:
> START_MONITORING_CONTAINER
> 2011-12-29 15:49:16,289 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Failed to launch container
> java.io.FileNotFoundException: File
> /tmp/nm-local-dir/usercache/jiangbing/appcache/application_1325062142731_0006
> does not exist
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:815)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:700)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:697)
> at
> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:697)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:123)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:237)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:67)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2011-12-29 15:49:16,290 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerExitEvent.EventType:
> CONTAINER_EXITED_WITH_FAILURE
> 2011-12-29 15:49:16,290 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Processing container_1325062142731_0006_01_000001 of type
> CONTAINER_EXITED_WITH_FAILURE
> 2011-12-29 15:49:16,290 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> Container container_1325062142731_0006_01_000001 transitioned from RUNNING
> to EXITED_WITH_FAILURE
> 2011-12-29 15:49:16,290 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEvent.EventType:
> CLEANUP_CONTAINER
> 2011-12-29 15:49:16,290 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Cleaning up container container_1325062142731_0006_01_000001
> 2011-12-29 15:49:16,290 DEBUG
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Marking container container_1325062142731_0006_01_000001 as inactive
> 2011-12-29 15:49:16,290 DEBUG
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Getting pid for container container_1325062142731_0006_01_000001 to kill
> from pid file
> /tmp/nm-local-dir/nmPrivate/container_1325062142731_0006_01_000001.pid
> 2011-12-29 15:49:16,290 DEBUG
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
> Accessing pid for container container_1325062142731_0006_01_000001 from pid
> file /tmp/nm-local-dir/nmPrivate/container_1325062142731_0006_01_000001.pid
> 2011-12-29 15:49:16,307 DEBUG
> org.apache.hadoop.yarn.event.AsyncDispatcher: Dispatching the event
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ContainerLocalizationCleanupEvent.EventType:
> CLEANUP_CONTAINER_RESOURCES
>
> In order to figure out the fact, I trace back to source code. I find that
> *org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor*:
>
> @Override
> public int launchContainer(Container container,
> Path nmPrivateContainerScriptPath, Path nmPrivateTokensPath,
> String userName, String appId, Path containerWorkDir)
> throws IOException {
> ....
> String[] sLocalDirs = getConf().getStrings(
> YarnConfiguration.NM_LOCAL_DIRS,
> YarnConfiguration.DEFAULT_NM_
> LOCAL_DIRS);
> for (String sLocalDir : sLocalDirs) {
> Path usersdir = new Path(sLocalDir, ContainerLocalizer.USERCACHE);
> Path userdir = new Path(usersdir, userName);
> Path appCacheDir = new Path(userdir, ContainerLocalizer.APPCACHE);
> Path appDir = new Path(appCacheDir, appIdStr);
> Path containerDir = new Path(appDir, containerIdStr);
> lfs.mkdir(containerDir, null, false);
> }
> ....
>
> lfs.mkdir(containerDir, null, false); refer to the api of mkdir, false
> means cannot create parent path here if not exists.
> In my hadoop project, I revise lfs.mkdir(containerDir, null, false); to
> lfs.mkdir(containerDir, null, true); , then my program goes well.
>
>
>
> 2012/1/9 raghavendhra rahul <ra...@gmail.com>
>
>> Any solutions.....
>>
>>
>> On Fri, Jan 6, 2012 at 5:15 PM, raghavendhra rahul <
>> raghavendhrarahul@gmail.com> wrote:
>>
>>> Hi all,
>>> I am trying to write an application master.Is there a way to specify
>>> node1: 10 conatiners
>>> node2: 10 containers
>>> Can we specify this kind of list using the application master????
>>>
>>> Also i set request.setHostName("client"); where client is the hostname
>>> of a node
>>> I checked the log to find the following error
>>> java.io.FileNotFoundException: File
>>> /local1/yarn/.yarn/local/usercache/rahul_2011/appcache/
>>> application_1325760852770_0001 does not exist
>>> at
>>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
>>> at
>>> org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:815)
>>> at
>>> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
>>> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
>>> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:700)
>>> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:697)
>>> at
>>> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
>>> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:697)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:122)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:237)
>>> at
>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:67)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> at java.lang.Thread.run(Thread.java:636)
>>>
>>> i.e containers are only launching within the host where the application
>>> master runs while the other nodes always remain free.
>>> Any ideas.....
>>>
>>
>>
>
>
> --
> Bing Jiang
> Tel：(86)134-2619-1361
> National Research Center for Intelligent Computing Systems
> Institute of Computing technology
> Graduate University of Chinese Academy of Science
>
>
Re: Container Launching
Posted by Bing Jiang <ji...@gmail.com>.
Hi, I have encountered a same question. Look below, and hope it can help u.
I get help from Hadoop-Yarn to deploy my real-time distributed computation
system, and I get reply from mapreduce-user@hadoop.apache.org to follow
these guilders below:
http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html
>>
>> http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
>>
> When I follow the steps to construct my Client, ApplicationMaster.And an
issue occurs to me that NM fail to launch a Container because of
java.io.FileNotFoundException.
So the part of NM log has been attached below:
....
2011-12-29 15:49:16,250 INFO org.apache.hadoop.yarn.server.
nodemanager.containermanager.application.Application: Adding
container_1325062142731_0006_01_000001 to application
application_1325062142731_0006
2011-12-29 15:49:16,250 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ApplicationLocalizationEvent.EventType:
INIT_APPLICATION_RESOURCES
2011-12-29 15:49:16,250 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationInitedEvent.EventType:
APPLICATION_INITED
2011-12-29 15:49:16,250 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Processing application_1325062142731_0006 of type APPLICATION_INITED
2011-12-29 15:49:16,250 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1325062142731_0006 transitioned from INITING to
RUNNING
2011-12-29 15:49:16,250 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerAppStartedEvent.EventType:
APPLICATION_STARTED
2011-12-29 15:49:16,250 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerInitEvent.EventType:
INIT_CONTAINER
2011-12-29 15:49:16,250 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Processing container_1325062142731_0006_01_000001 of type INIT_CONTAINER
2011-12-29 15:49:16,250 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1325062142731_0006_01_000001 transitioned from NEW to
LOCALIZED
2011-12-29 15:49:16,250 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEvent.EventType:
LAUNCH_CONTAINER
2011-12-29 15:49:16,287 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEvent.EventType:
CONTAINER_LAUNCHED
2011-12-29 15:49:16,287 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Processing container_1325062142731_0006_01_000001 of type CONTAINER_LAUNCHED
2011-12-29 15:49:16,287 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1325062142731_0006_01_000001 transitioned from
LOCALIZED to RUNNING
2011-12-29 15:49:16,288 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerStartMonitoringEvent.EventType:
START_MONITORING_CONTAINER
2011-12-29 15:49:16,289 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Failed to launch container
java.io.FileNotFoundException: File
/tmp/nm-local-dir/usercache/jiangbing/appcache/application_1325062142731_0006
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:815)
at
org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:700)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:697)
at
org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:697)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:123)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:237)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:67)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2011-12-29 15:49:16,290 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerExitEvent.EventType:
CONTAINER_EXITED_WITH_FAILURE
2011-12-29 15:49:16,290 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Processing container_1325062142731_0006_01_000001 of type
CONTAINER_EXITED_WITH_FAILURE
2011-12-29 15:49:16,290 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1325062142731_0006_01_000001 transitioned from RUNNING
to EXITED_WITH_FAILURE
2011-12-29 15:49:16,290 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEvent.EventType:
CLEANUP_CONTAINER
2011-12-29 15:49:16,290 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_1325062142731_0006_01_000001
2011-12-29 15:49:16,290 DEBUG
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Marking container container_1325062142731_0006_01_000001 as inactive
2011-12-29 15:49:16,290 DEBUG
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Getting pid for container container_1325062142731_0006_01_000001 to kill
from pid file
/tmp/nm-local-dir/nmPrivate/container_1325062142731_0006_01_000001.pid
2011-12-29 15:49:16,290 DEBUG
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Accessing pid for container container_1325062142731_0006_01_000001 from pid
file /tmp/nm-local-dir/nmPrivate/container_1325062142731_0006_01_000001.pid
2011-12-29 15:49:16,307 DEBUG org.apache.hadoop.yarn.event.AsyncDispatcher:
Dispatching the event
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.ContainerLocalizationCleanupEvent.EventType:
CLEANUP_CONTAINER_RESOURCES
In order to figure out the fact, I trace back to source code. I find that *
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor*:
@Override
public int launchContainer(Container container,
Path nmPrivateContainerScriptPath, Path nmPrivateTokensPath,
String userName, String appId, Path containerWorkDir)
throws IOException {
....
String[] sLocalDirs = getConf().getStrings(
YarnConfiguration.NM_LOCAL_DIRS,
YarnConfiguration.DEFAULT_NM_
LOCAL_DIRS);
for (String sLocalDir : sLocalDirs) {
Path usersdir = new Path(sLocalDir, ContainerLocalizer.USERCACHE);
Path userdir = new Path(usersdir, userName);
Path appCacheDir = new Path(userdir, ContainerLocalizer.APPCACHE);
Path appDir = new Path(appCacheDir, appIdStr);
Path containerDir = new Path(appDir, containerIdStr);
lfs.mkdir(containerDir, null, false);
}
....
lfs.mkdir(containerDir, null, false); refer to the api of mkdir, false
means cannot create parent path here if not exists.
In my hadoop project, I revise lfs.mkdir(containerDir, null, false); to
lfs.mkdir(containerDir, null, true); , then my program goes well.
2012/1/9 raghavendhra rahul <ra...@gmail.com>
> Any solutions.....
>
>
> On Fri, Jan 6, 2012 at 5:15 PM, raghavendhra rahul <
> raghavendhrarahul@gmail.com> wrote:
>
>> Hi all,
>> I am trying to write an application master.Is there a way to specify
>> node1: 10 conatiners
>> node2: 10 containers
>> Can we specify this kind of list using the application master????
>>
>> Also i set request.setHostName("client"); where client is the hostname of
>> a node
>> I checked the log to find the following error
>> java.io.FileNotFoundException: File
>> /local1/yarn/.yarn/local/usercache/rahul_2011/appcache/
>> application_1325760852770_0001 does not exist
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
>> at
>> org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:815)
>> at
>> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
>> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
>> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:700)
>> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:697)
>> at
>> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
>> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:697)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:122)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:237)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:67)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> i.e containers are only launching within the host where the application
>> master runs while the other nodes always remain free.
>> Any ideas.....
>>
>
>
--
Bing Jiang
Tel：(86)134-2619-1361
National Research Center for Intelligent Computing Systems
Institute of Computing technology
Graduate University of Chinese Academy of Science
Re: Container Launching
Posted by raghavendhra rahul <ra...@gmail.com>.
Any solutions.....
On Fri, Jan 6, 2012 at 5:15 PM, raghavendhra rahul <
raghavendhrarahul@gmail.com> wrote:
> Hi all,
> I am trying to write an application master.Is there a way to specify
> node1: 10 conatiners
> node2: 10 containers
> Can we specify this kind of list using the application master????
>
> Also i set request.setHostName("client"); where client is the hostname of
> a node
> I checked the log to find the following error
> java.io.FileNotFoundException: File
> /local1/yarn/.yarn/local/usercache/rahul_2011/appcache/
> application_1325760852770_0001 does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
> at
> org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:815)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:700)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:697)
> at
> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:697)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:122)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:237)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:67)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
>
> i.e containers are only launching within the host where the application
> master runs while the other nodes always remain free.
> Any ideas.....
>
Re: Container Launching
Posted by raghavendhra rahul <ra...@gmail.com>.
Any solutions.....
On Fri, Jan 6, 2012 at 5:15 PM, raghavendhra rahul <
raghavendhrarahul@gmail.com> wrote:
> Hi all,
> I am trying to write an application master.Is there a way to specify
> node1: 10 conatiners
> node2: 10 containers
> Can we specify this kind of list using the application master????
>
> Also i set request.setHostName("client"); where client is the hostname of
> a node
> I checked the log to find the following error
> java.io.FileNotFoundException: File
> /local1/yarn/.yarn/local/usercache/rahul_2011/appcache/
> application_1325760852770_0001 does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431)
> at
> org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:815)
> at
> org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
> at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:700)
> at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:697)
> at
> org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
> at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:697)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:122)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:237)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:67)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
>
> i.e containers are only launching within the host where the application
> master runs while the other nodes always remain free.
> Any ideas.....
>