Container exited with a non-zero exit code

[Zhudacai <zh...@hisilicon.com>]

Hi.all,

I've just done a fresh install of Hadoop with three nodes, one master (NameNode, SecondNameNode, ResourceTracker) and two slaves (DataNode). The HDFS are successfully formatted, all services are up. When I run the examples, e.g. teragen, terasort, I occasionally got this exception:

/15/07/23 19:55:34 INFO mapreduce.Job: map 0% reduce 0%//
//15/07/23 19:55:40 INFO mapreduce.Job: Task Id : attempt_1437652487249_0001_m_000000_0, Status : FAILED//
//Exception from container-launch.//
//Container id: container_1437652487249_0001_01_000002//
//Exit code: 134//
//Exception message: /bin/bash: line 1: 21736 Aborted //
///usr/openjdk-1.8.0-internal/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m //
//-Djava.io.tmpdir=/home/hadoop3/tmp/nm-local-dir/usercache/root/appcache/application_1437652487249_0001/container_1437652487249_0001_01_000002/tmp //
//-Dlog4j.configuration=container-log4j.properties //
//-Dyarn.app.container.log.dir=/home/hadoop3/hadoop-2.6.0/logs/userlogs/application_1437652487249_0001/container_1437652487249_0001_01_000002 //
//-Dyarn.app.container.log.filesize=0 //
//-Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.1.9 39868 attempt_1437652487249_0001_m_000000_0 2 > //
///home/hadoop3/hadoop-2.6.0/logs/userlogs/application_1437652487249_0001/container_1437652487249_0001_01_000002/stdout 2> /home/hadoop3/hadoop-2.6.0/logs/userlogs/application_1437652487249_0001/container_1437652487249_0001_01_000002/stderr//
//
//Stack trace: ExitCodeException exitCode=134: /bin/bash: line 1: 21736 Aborted //
///usr/openjdk-1.8.0-internal/bin/java -Djava.net.preferIPv4Stack=true //
//-Dhadoop.metrics.log.level=WARN -Xmx200m -Djava.io.tmpdir=/home/hadoop3/tmp/nm-local-dir/usercache/root/appcache/application_1437652487249_0001/container_1437652487249_0001_01_000002/tmp //
//-Dlog4j.configuration=container-log4j.properties //
//-Dyarn.app.container.log.dir=/home/hadoop3/hadoop-2.6.0/logs/userlogs/application_1437652487249_0001/container_1437652487249_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 192.168.1.9 39868 attempt_1437652487249_0001_m_000000_0 2 > /home/hadoop3/hadoop-2.6.0/logs/userlogs/application_1437652487249_0001/container_1437652487249_0001_01_000002/stdout 2> /home/hadoop3/hadoop-2.6.0/logs/userlogs/application_1437652487249_0001/container_1437652487249_0001_01_000002/stderr//
//
//at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)//
//at org.apache.hadoop.util.Shell.run(Shell.java:455)//
//at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)//
//at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)//
//at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)//
//at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)//
//at java.util.concurrent.FutureTask.run(FutureTask.java:266)//
//at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)//
//at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)//
//at java.lang.Thread.run(Thread.java:745)//
//
//
//Container exited with a non-zero exit code 134//
//
//15/07/23 19:55:46 INFO mapreduce.Job: map 3% reduce 0%/

But the job could succfully completed.
Here're the configurations:

core-site.xml
/<configuration>//
//<property>//
//<name>fs.defaultFS</name>//
//<value>hdfs://master:9000</value>//
//<final>true</final>//
//</property>//
//<property>//
//<name>hadoop.tmp.dir</name>//
//<value>file:/home/hadoop3/tmp</value>//
//</property>//
//</configuration>//
//
//hdfs-site.xml//
//<configuration>//
//<property>//
//<name>dfs.namenode.secondary.http-address</name>//
//<value>master:50090</value>//
//</property>//
//<property>//
//<name>dfs.namenode.name.dir</name>//
//<value>file:/home/hadoop3/tmp/dfs/name</value>//
//</property>//
//<property>//
//<name>dfs.datanode.data.dir</name>//
//<value>file:/home/hadoop3/tmp/dfs/data</value>//
//</property>//
//<property>//
//<name>dfs.replication</name>//
//<value>1</value>//
//</property>//
//</configuration>//
/
yarn-site.xml
/<configuration>//
//<property>//
//<name>yarn.resourcemanager.hostname</name>//
//<value>master</value>//
//</property>//
//<property>//
//<name>yarn.nodemanager.aux-services</name>//
//<value>mapreduce_shuffle</value>//
//</property>//
//<property>//
//<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>//
//<value>org.apache.hadoop.mapred.ShuffleHandler</value>//
//</property>//
//<property>//
//<name>yarn.resourcemanager.address</name>//
//<value>master:8032</value>//
//</property>//
//<property>//
//<name>yarn.resourcemanager.scheduler.address</name>//
//<value>master:8030</value>//
//</property>//
//<property>//
//<name>yarn.resourcemanager.resource-tracker.address</name>//
//<value>master:8035</value>//
//</property>//
//<property>//
//<name>yarn.resourcemanager.admin.address</name>//
//<value>master:8033</value>//
//</property>//
//<property>//
//<name>yarn.resourcemanager.webapp.address</name>//
//<value>master:8088</value>//
//</property>//
//<property>//
//<name>yarn.nodemanager.resource.cpu-vcores</name>//
//<value>16</value>//
//</property>//
//</configuration>/

mapred-site.xml
/<configuration>//
//<property>//
//<name>mapreduce.framework.name</name>//
//<value>yarn</value>//
//</property>//
//</configuration>/

I also noticed if hadoop was installed on single node, the exception would never show up.
The number of the exception increases with the values of -Dmapred.map.tasks and -Dmapred.reduce.tasks.

I'm using hadoop 2.6.0, OpenJDK 1.8, runing on the arm64 platform.

Best Regards

Jared