You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Soni,Mohit" <mo...@gmail.com> on 2014/05/30 19:19:27 UTC
cgroups: Failed to initialize container executor
Hi
I am trying to use cgroups with YARN 2.4.0 single-node setup running on Ubuntu 13.10. It works fine if I use DefaultContainerExecutor. But, I get error when I try to use LinuxContainerExecutor. I referred to http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SecureMode.html#LinuxContainerExecutor for help.
My setup:
User: hduser
Group: hadoop
Permissions for bin directory:
hduser@host1:/usr/local/hadoop/bin$ ls -l
total 364
---Sr-s--- 1 root hadoop 85670 May 30 00:14 container-executor
-r-------- 1 root hadoop 97 May 30 00:52 container-executor.cfg
-rw-r--r-- 1 hduser hadoop 71 May 30 00:44 container-executor.cfg.orig
-rwxr-xr-x 1 hduser hadoop 85670 May 30 00:22 container-executor.orig
-rwxr-xr-x 1 hduser hadoop 4857 Mar 31 08:49 hadoop
-rwxr-xr-x 1 hduser hadoop 7541 Mar 31 08:49 hadoop.cmd
-rwxr-xr-x 1 hduser hadoop 8200 Mar 31 08:49 hdfs
-rwxr-xr-x 1 hduser hadoop 6162 Mar 31 08:49 hdfs.cmd
-rwxr-xr-x 1 hduser hadoop 5205 Mar 31 08:49 mapred
-rwxr-xr-x 1 hduser hadoop 5559 Mar 31 08:49 mapred.cmd
-rwxr-xr-x 1 hduser hadoop 1776 Mar 31 08:49 rcc
-rwxr-xr-x 1 hduser hadoop 108825 Mar 31 08:49 test-container-executor
-rwxr-xr-x 1 hduser hadoop 9539 Mar 31 08:49 yarn
-rwxr-xr-x 1 hduser hadoop 9988 Mar 31 08:49 yarn.cmd
File: container-executor.cfg
yarn.nodemanager.linux-container-executor.group=hadoop
min.user.id=1
allowed.system.users=hduser
File: yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<!-- Cgroups -->
<property>
<description>who will execute(launch) the containers.</description>
<name>yarn.nodemanager.container-executor.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property>
<property>
<description>The class which should help the LCE handle resources.</description>
<name>yarn.nodemanager.linux-container-executor.resources-handler.class</name>
<value>org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
<value>/hadoop-yarn</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
<value>/sys/fs/cgroup</value>
</property>
<property>
<name>yarn.nodemanager.linux-container-executor.group</name>
<value>hadoop</value>
</property>
Now, when I execute $HADOOP_INSTALL/bin/yarn nodemanager, I fails with exit code 255, with following stacktrace:
14/05/30 01:10:10 INFO nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT]
14/05/30 01:10:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/05/30 01:10:11 INFO service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:144)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:357)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:404)
Caused by: java.io.IOException: Cannot run program "/usr/local/hadoop/bin/container-executor": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:448)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:169)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:142)
... 3 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
... 8 more
14/05/30 01:10:11 FATAL nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:144)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:357)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:404)
Caused by: java.io.IOException: Cannot run program "/usr/local/hadoop/bin/container-executor": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:448)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:169)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:142)
... 3 more
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:135)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
... 8 more
14/05/30 01:10:11 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
It would be great if someone can please help me figure out what’s wrong with my setup.
Thanks
Mohit