You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by iain wright <ia...@gmail.com> on 2014/09/29 20:56:49 UTC
YARN application failing to localize needed jars (Xpost: hbase users)
Hi folks,
I'm having trouble using HBASE copyTable to seed an existing tables data to
a replication peer. Surely its an oversight in configuration on our part,
but I've scoured the web and doc's for a couple days now.
We have been able to run these jobs with success (perhaps they don't
require localization of any jars since the jars are already included in the
environment on each nodemanager?):
./hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -fileSize 5GB
./hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 5
When running copyTable it seems like the resources are placed on the yarn
application manager but not localized to the nodemanagers.
I've tried several settings for yarn.app.mapreduce.am.staging-dir and in
the below iteration im trying the defaults, the path exists with correct
permissions on each NM/AM
hadoop.tmp.dir is specified in core-site.xml
Here is the full command we are running:
[hbase@master2 bin]$ ./hbase org.apache.hadoop.hbase.mapreduce.CopyTable
--starttime=1409128964 --peer.adr=master0.hbasex1.test.cloud.domain.tv,
master1.hbasex1.test.cloud.v.tv,master2.hbasex1.test.cloud.domain.tv:2181:/hbase
moderation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/hbase-0.98.6.1-1a/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/hadoop-2.5.0-1a/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
2014-09-26 23:51:24,141 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2014-09-26 23:51:24,863 WARN [main] shortcircuit.DomainSocketFactory: The
short-circuit local reads feature cannot be used because libhadoop cannot
be loaded.
2014-09-26 23:51:24,910 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.HConstants, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,911 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.client.Put, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.CompatibilityFactory, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,913 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.zookeeper.ZooKeeper, using jar
/usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar
2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.jboss.netty.channel.ChannelFactory, using jar
/usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar
2014-09-26 23:51:24,915 DEBUG [main] mapreduce.TableMapReduceUtil: For
class com.google.protobuf.Message, using jar
/usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar
2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For
class com.google.common.collect.Lists, using jar
/usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar
2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.cloudera.htrace.Trace, using jar
/usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar
2014-09-26 23:51:24,917 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.cliffc.high_scale_lib.Counter, using jar
/usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar
2014-09-26 23:51:24,921 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.io.LongWritable, using jar
/usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,922 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.io.Text, using jar
/usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.io.LongWritable, using jar
/usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,924 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.io.Text, using jar
/usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,925 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar
/usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar
2014-09-26 23:51:24,926 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar
/usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar
2014-09-26 23:51:24,952 INFO [main] Configuration.deprecation:
io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.HConstants, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.client.Put, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.CompatibilityFactory, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,965 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.zookeeper.ZooKeeper, using jar
/usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar
2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.jboss.netty.channel.ChannelFactory, using jar
/usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar
2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For
class com.google.protobuf.Message, using jar
/usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar
2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For
class com.google.common.collect.Lists, using jar
/usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar
2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.cloudera.htrace.Trace, using jar
/usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar
2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.cliffc.high_scale_lib.Counter, using jar
/usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar
2014-09-26 23:51:24,971 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.io.Writable, using jar
/usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.io.Writable, using jar
/usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
2014-09-26 23:51:24,974 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.hbase.mapreduce.TableOutputFormat, using jar
/usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
2014-09-26 23:51:24,975 DEBUG [main] mapreduce.TableMapReduceUtil: For
class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar
/usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar
2014-09-26 23:51:25,035 INFO [main] client.RMProxy: Connecting to
ResourceManager at master2.hbase1a.test.cloud.domain.tv/172.16.0.145:8032
2014-09-26 23:51:25,191 INFO [main] Configuration.deprecation:
io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-09-26 23:51:25,247 INFO [main] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x6bf3ede0 connecting to ZooKeeper ensemble=
master1.hbasex1.test.cloud.domain.tv:2181,
master0.hbasex1.test.cloud.domain.tv:2181,
master2.hbasex1.test.cloud.domain.tv:2181
2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client
environment:host.name=master2.hbase1a.test.cloud.domain.tv
2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_65
2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre
2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client
environment:java.class.path=/usr/local/hbase/bin/../conf:/usr/lib/jvm/java/lib/tools.jar:/usr/local/hbase/bin/..:/usr/local/hbase/bin/../lib/activation-1.1.jar:/usr/local/hbase/bin/../lib/aopalliance-1.0.jar:/usr/local/hbase/bin/../lib/asm-3.1.jar:/usr/local/hbase/bin/../lib/avro-1.7.4.jar:/usr/local/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/bin/../lib/commons-cli-1.2.jar:/usr/local/hbase/bin/../lib/commons-codec-1.7.jar:/usr/local/hbase/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase/bin/../lib/commons-compress-1.4.1.jar:/usr/local/hbase/bin/../lib/commons-configuration-1.6.jar:/usr/local/hbase/bin/../lib/commons-daemon-1.0.13.jar:/usr/local/hbase/bin/../lib/commons-digester-1.8.jar:/usr/local/hbase/bin/../lib/commons-el-1.0.jar:/usr/local/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/local/hbase/bin/../lib/commons-io-2.4.jar:/usr/local/hbase/bin/../lib/commons-lang-2.6.jar:/usr/local/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hbase/bin/../lib/commons-math-2.1.jar:/usr/local/hbase/bin/../lib/commons-net-3.1.jar:/usr/local/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hbase/bin/../lib/gmbal-api-only-3.0.0-b023.jar:/usr/local/hbase/bin/../lib/grizzly-framework-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-server-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-servlet-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-rcm-2.1.2.jar:/usr/local/hbase/bin/../lib/guava-12.0.1.jar:/usr/local/hbase/bin/../lib/guice-3.0.jar:/usr/local/hbase/bin/../lib/guice-servlet-3.0.jar:/usr/local/hbase/bin/../lib/hamcrest-core-1.3.jar:/usr/local/hbase/bin/../lib/hbase-client-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-examples-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop2-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-prefix-tree-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-protocol-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-shell-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-testing-util-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-thrift-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/bin/../lib/htrace-core-2.04.jar:/usr/local/hbase/bin/../lib/httpclient-4.1.3.jar:/usr/local/hbase/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase/bin/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-xc-1.8.8.jar:/usr/local/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/bin/../lib/javax.inject-1.jar:/usr/local/hbase/bin/../lib/javax.servlet-3.1.jar:/usr/local/hbase/bin/../lib/javax.servlet-api-3.0.1.jar:/usr/local/hbase/bin/../lib/jaxb-api-2.2.2.jar:/usr/local/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/bin/../lib/jersey-client-1.9.jar:/usr/local/hbase/bin/../lib/jersey-core-1.8.jar:/usr/local/hbase/bin/../lib/jersey-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jersey-guice-1.9.jar:/usr/local/hbase/bin/../lib/jersey-json-1.8.jar:/usr/local/hbase/bin/../lib/jersey-server-1.8.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-core-1.9.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jets3t-0.6.1.jar:/usr/local/hbase/bin/../lib/jettison-1.3.1.jar:/usr/local/hbase/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase/bin/../lib/jruby-complete-1.6.8.jar:/usr/local/hbase/bin/../lib/jsch-0.1.42.jar:/usr/local/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsr305-1.3.9.jar:/usr/local/hbase/bin/../lib/junit-4.11.jar:/usr/local/hbase/bin/../lib/libthrift-0.9.0.jar:/usr/local/hbase/bin/../lib/log4j-1.2.17.jar:/usr/local/hbase/bin/../lib/management-api-3.0.0-b012.jar:/usr/local/hbase/bin/../lib/metrics-core-2.2.0.jar:/usr/local/hbase/bin/../lib/netty-3.6.6.Final.jar:/usr/local/hbase/bin/../lib/paranamer-2.3.jar:/usr/local/hbase/bin/../lib/protobuf-java-2.5.0.jar:/usr/local/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase/bin/../lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hbase/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase/bin/../lib/xz-1.0.jar:/usr/local/hbase/bin/../lib/zookeeper-3.4.6.jar:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/etc/hadoop/
2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/local/hadoop/lib
2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux
2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client
environment:os.version=3.10.42-52.145.amzn1.x86_64
2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client
environment:user.name=hbase
2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client
environment:user.home=/home/hbase
2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client
environment:user.dir=/usr/local/hbase-0.98.6.1-1a/bin
2014-09-26 23:51:25,254 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=master1.hbasex1.test.cloud.domain.tv:2181,
master0.hbasex1.test.cloud.domain.tv:2181,
master2.hbasex1.test.cloud.domain.tv:2181 sessionTimeout=90000
watcher=hconnection-0x6bf3ede0, quorum=
master1.hbasex1.test.cloud.domain.tv:2181,
master0.hbasex1.test.cloud.domain.tv:2181,
master2.hbasex1.test.cloud.domain.tv:2181, baseZNode=/hbase
2014-09-26 23:51:25,270 INFO [main-SendThread(
master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181. Will not attempt to
authenticate using SASL (unknown error)
2014-09-26 23:51:25,276 INFO [main-SendThread(
master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket
connection established to
master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, initiating session
2014-09-26 23:51:25,282 INFO [main-SendThread(
master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, sessionid =
0x248afb6109b007f, negotiated timeout = 40000
2014-09-26 23:51:25,308 INFO [main] mapreduce.TableOutputFormat: Created
table instance for moderation
2014-09-26 23:51:25,469 INFO [main] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0xa7a371 connecting to ZooKeeper ensemble=
master1.hbase1a.test.cloud.domain.tv:2181,
master0.hbase1a.test.cloud.domain.tv:2181,
master2.hbase1a.test.cloud.domain.tv:2181
2014-09-26 23:51:25,469 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=master1.hbase1a.test.cloud.domain.tv:2181,
master0.hbase1a.test.cloud.domain.tv:2181,
master2.hbase1a.test.cloud.domain.tv:2181 sessionTimeout=90000
watcher=hconnection-0xa7a371, quorum=
master1.hbase1a.test.cloud.domain.tv:2181,
master0.hbase1a.test.cloud.domain.tv:2181,
master2.hbase1a.test.cloud.domain.tv:2181, baseZNode=/hbase
2014-09-26 23:51:25,470 INFO [main-SendThread(
master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Opening
socket connection to server
master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181. Will not attempt to
authenticate using SASL (unknown error)
2014-09-26 23:51:25,470 INFO [main-SendThread(
master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket
connection established to
master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, initiating session
2014-09-26 23:51:25,472 INFO [main-SendThread(
master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Session
establishment complete on server
master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, sessionid =
0x34877671747028e, negotiated timeout = 40000
2014-09-26 23:51:25,490 INFO [main] util.RegionSizeCalculator: Calculating
region sizes for table "moderation".
2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region
moderation,,1411774704618.68a1839eacfa355028522286caaf7cd5. has size 0
2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region
sizes calculated
2014-09-26 23:51:25,709 DEBUG [main] client.ClientSmallScanner: Finished
with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
STARTKEY => '', ENDKEY => ''}
2014-09-26 23:51:25,722 DEBUG [main] mapreduce.TableInputFormatBase:
getSplits: split -> 0 -> ip-172-16-0-247.us-west-1.compute.internal:,
2014-09-26 23:51:25,739 INFO [main] mapreduce.JobSubmitter: number of
splits:1
2014-09-26 23:51:25,748 INFO [main] Configuration.deprecation:
io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-09-26 23:51:25,895 INFO [main] mapreduce.JobSubmitter: Submitting
tokens for job: job_1411771844660_0004
2014-09-26 23:51:26,033 INFO [main] impl.YarnClientImpl: Submitted
application application_1411771844660_0004
2014-09-26 23:51:26,066 INFO [main] mapreduce.Job: The url to track the
job:
http://master2.hbase1a.test.cloud.domain.tv:8088/proxy/application_1411771844660_0004/
2014-09-26 23:51:26,067 INFO [main] mapreduce.Job: Running job:
job_1411771844660_0004
2014-09-26 23:51:29,082 INFO [main] mapreduce.Job: Job
job_1411771844660_0004 running in uber mode : false
2014-09-26 23:51:29,084 INFO [main] mapreduce.Job: map 0% reduce 0%
2014-09-26 23:51:29,094 INFO [main] mapreduce.Job: Job
job_1411771844660_0004 failed with state FAILED due to: Application
application_1411771844660_0004 failed 2 times due to AM Container for
appattempt_1411771844660_0004_000002 exited with exitCode: -1000 due to:
File
file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar
does not exist
.Failing this attempt.. Failing the application.
2014-09-26 23:51:29,111 INFO [main] mapreduce.Job: Counters: 0
The directory on the AM (master) from which the command is initiated as
hbase user:
[hbase@master2 bin]$ ls -lah
/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/
total 3.7M
drwx------ 3 hbase hbase 4.0K Sep 26 23:51 .
drwx------ 10 hbase hadoop 4.0K Sep 26 23:51 ..
-rw-r--r-- 1 hbase hbase 3.5M Sep 26 23:51 job.jar
-rw-rw-r-- 1 hbase hbase 28K Sep 26 23:51 .job.jar.crc
-rw-r--r-- 1 hbase hbase 111 Sep 26 23:51 job.split
-rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.split.crc
-rw-r--r-- 1 hbase hbase 56 Sep 26 23:51 job.splitmetainfo
-rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.splitmetainfo.crc
-rw-r--r-- 1 hbase hbase 102K Sep 26 23:51 job.xml
-rw-rw-r-- 1 hbase hbase 824 Sep 26 23:51 .job.xml.crc
drwx------ 2 hbase hbase 4.0K Sep 26 23:51 libjars
The same directory is empty on each NM.
Heres also the log from a node that got the job/application:
2014-09-26 23:51:26,635 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
Auth successful for appattempt_1411771844660_0004_000001 (auth:SIMPLE)
2014-09-26 23:51:26,639 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Start request for container_1411771844660_0004_01_000001 by user hbase
2014-09-26 23:51:26,639 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Creating a new application reference for app application_1411771844660_0004
2014-09-26 23:51:26,639 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase
IP=172.16.0.145 OPERATION=Start Container Request
TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1411771844660_0004
CONTAINERID=container_1411771844660_0004_01_000001
2014-09-26 23:51:26,639 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1411771844660_0004 transitioned from NEW to INITING
2014-09-26 23:51:26,639 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Adding container_1411771844660_0004_01_000001 to application
application_1411771844660_0004
2014-09-26 23:51:26,639 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1411771844660_0004 transitioned from INITING to
RUNNING
2014-09-26 23:51:26,640 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1411771844660_0004_01_000001 transitioned from NEW to
LOCALIZING
2014-09-26 23:51:26,640 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_INIT for appId application_1411771844660_0004
2014-09-26 23:51:26,640 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar
transitioned from INIT to DOWNLOADING
2014-09-26 23:51:26,640 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.splitmetainfo
transitioned from INIT to DOWNLOADING
2014-09-26 23:51:26,640 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.split
transitioned from INIT to DOWNLOADING
2014-09-26 23:51:26,641 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.xml
transitioned from INIT to DOWNLOADING
2014-09-26 23:51:26,641 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Created localizer for container_1411771844660_0004_01_000001
2014-09-26 23:51:26,645 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Writing credentials to the nmPrivate file
/data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens.
Credentials list:
2014-09-26 23:51:26,648 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Initializing user hbase
2014-09-26 23:51:26,661 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying
from
/data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens
to
/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001.tokens
2014-09-26 23:51:26,661 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set
to
/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004
=
file:/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004
2014-09-26 23:51:26,696 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
DEBUG: FAILED {
file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar,
1411775485000, PATTERN, (?:classes/|lib/).* }, File
file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar
does not exist
2014-09-26 23:51:26,697 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
Resource
file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar(->/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/filecache/10/job.jar)
transitioned from DOWNLOADING to FAILED
2014-09-26 23:51:26,697 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1411771844660_0004_01_000001 transitioned from
LOCALIZING to LOCALIZATION_FAILED
2014-09-26 23:51:26,697 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
Container container_1411771844660_0004_01_000001 sent RELEASE event on a
resource request {
file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar,
1411775485000, PATTERN, (?:classes/|lib/).* } not present in cache.
2014-09-26 23:51:26,698 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path :
/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001
2014-09-26 23:51:26,698 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
Unknown localizer with localizerId container_1411771844660_0004_01_000001
is sending heartbeat. Ordering it to DIE
2014-09-26 23:51:26,698 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: delete
returned false for path:
[/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001]
2014-09-26 23:51:26,698 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path :
/data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001
2014-09-26 23:51:26,699 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: delete
returned false for path:
[/data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001]
2014-09-26 23:51:26,699 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase
OPERATION=Container Finished - Failed TARGET=ContainerImpl
RESULT=FAILURE DESCRIPTION=Container failed with state:
LOCALIZATION_FAILED APPID=application_1411771844660_0004
CONTAINERID=container_1411771844660_0004_01_000001
2014-09-26 23:51:26,699 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1411771844660_0004_01_000001 transitioned from
LOCALIZATION_FAILED to DONE
2014-09-26 23:51:26,699 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Removing container_1411771844660_0004_01_000001 from application
application_1411771844660_0004
2014-09-26 23:51:26,699 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event CONTAINER_STOP for appId application_1411771844660_0004
2014-09-26 23:51:27,629 INFO
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed
completed containers from NM context:
[container_1411771844660_0004_01_000001]
2014-09-26 23:51:28,252 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Stopping resource-monitoring for container_1411771844660_0004_01_000001
2014-09-26 23:51:29,631 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1411771844660_0004 transitioned from RUNNING to
APPLICATION_RESOURCES_CLEANINGUP
2014-09-26 23:51:29,632 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path :
/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004
2014-09-26 23:51:29,632 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path :
/data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004
2014-09-26 23:51:29,632 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got
event APPLICATION_STOP for appId application_1411771844660_0004
2014-09-26 23:51:29,632 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1411771844660_0004 transitioned from
APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2014-09-26 23:51:29,632 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler:
Scheduling Log Deletion for application: application_1411771844660_0004,
with delay of 10800 seconds
Thank you in advance for any advice
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.