You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Håvard Wahl Kongsgård <ha...@gmail.com> on 2015/10/05 16:54:12 UTC
Problem importing hbase exports
Hi, Iam having problems importing older hbase exports( hbase 0.94).
Hadoop/hbase fails with "java.io.IOException: keyvalues=NONE read 4096
bytes, should read 327381". What I have noticed is that the latest
version of hbase import splits the files into multiple parts/task,
while in hbase 0.9X import only created one job per file.
My current system is 1.0.0-cdh5.4.7.
command
sudo -u hbase hbase -Dhbase.import.version=0.94
org.apache.hadoop.hbase.mapreduce.Import photos
hdfs://rack3:8020/hbase_photos
Log
2015-10-05 16:31:55,005 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
2015-10-05 16:31:55,519 INFO
org.apache.hadoop.conf.Configuration.deprecation: session.id is
deprecated. Instead, use dfs.metrics.session-id
2015-10-05 16:31:55,520 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-cdh5.0.0-beta-1--1, built on
10/28/2013 00:00 GMT
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=rack3
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_67
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-7-oracle-cloudera/jre
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.class.path=/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle-cloudera/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.7.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/etc/hadoop/conf:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/slf4j-log4j12.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.5.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/.//hadoop-nfs.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-common-2.2.0-cdh5.0.0-beta-1-tests.jar:/usr/lib/hadoop/.//hadoop-annotations-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-nfs-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-auth-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.2.0-cdh5.0.0-beta-1-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.7.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-examples.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-test.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-test-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-tools-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-examples-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-tools.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core-2.2.0-mr1-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/jars/classes:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/jars/job.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-912292882399157935_1505303099_946479403/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-common-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-6963687099100849172_377036872_946479477/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-1572551158342880427_72556552_946479527/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/htrace-core-2.01.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/2785619607140856733_609359336_946479607/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-protocol-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/5634015324466447495_1643919977_946479677/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-2466065489594964804_983196214_946479753/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hadoop-core-2.2.0-mr1-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/5926457346568904278_1528333591_946479800/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/protobuf-java-2.5.0.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-9219457335404538790_2034210828_946479843/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-hadoop-compat-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/146573646801466628_-788370405_946479879/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-client-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/5567766410265772826_-1711486573_946480347/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-server-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/4939585511868799146_854929102_946480411/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/guava-12.0.1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/3116845128622493648_-856562511_946480494/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/netty-3.2.2.Final.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/attempt_201510051609_0001_m_000000_2/work
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/hadoop-0.20-mapreduce/lib/native/Linux-amd64-64:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/attempt_201510051609_0001_m_000000_2/work
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/attempt_201510051609_0001_m_000000_2/work/tmp
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.name=Linux
2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.arch=amd64
2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper: Client
environment:os.version=3.16.0-30-generic
2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.name=mapred
2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.home=/usr/lib/hadoop-0.20-mapreduce
2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper: Client
environment:user.dir=/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/attempt_201510051609_0001_m_000000_2/work
2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper:
Initiating client connection, connectString=localhost:2181
sessionTimeout=90000 watcher=hconnection-0x1d63530a
2015-10-05 16:31:56,051 INFO
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process
identifier=hconnection-0x1d63530a connecting to ZooKeeper
ensemble=localhost:2181
2015-10-05 16:31:56,053 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not
attempt to authenticate using SASL (unknown error)
2015-10-05 16:31:56,054 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating
session
2015-10-05 16:31:56,540 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server localhost/0:0:0:0:0:0:0:1:2181,
sessionid = 0x1503856871c001b, negotiated timeout = 40000
2015-10-05 16:31:57,087 INFO
org.apache.hadoop.hbase.mapreduce.TableOutputFormat: Created table
instance for photos
2015-10-05 16:31:57,096 INFO org.apache.hadoop.util.ProcessTree:
setsid exited with exit code 0
2015-10-05 16:31:57,100 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@404226a8
2015-10-05 16:31:57,416 INFO org.apache.hadoop.mapred.MapTask:
Processing split:
hdfs://0.0.0.0:8020/hbase_fb_photos_compressed/part-m-00033:2281701376+143727446
2015-10-05 16:31:57,456 INFO
org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2015-10-05 16:31:57,457 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new decompressor [.deflate]
2015-10-05 16:31:57,548 INFO org.apache.zookeeper.ZooKeeper:
Initiating client connection, connectString=localhost:2181
sessionTimeout=90000 watcher=attempt_201510051609_0001_m_000000_2
2015-10-05 16:31:57,549 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/127.0.0.1:2181. Will not attempt
to authenticate using SASL (unknown error)
2015-10-05 16:31:57,550 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to localhost/127.0.0.1:2181, initiating session
2015-10-05 16:31:57,550 INFO
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process
identifier=attempt_201510051609_0001_m_000000_2 connecting to
ZooKeeper ensemble=localhost:2181
2015-10-05 16:31:57,566 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server localhost/127.0.0.1:2181, sessionid =
0x1503856871c001d, negotiated timeout = 40000
2015-10-05 16:31:57,574 INFO org.apache.zookeeper.ZooKeeper: Session:
0x1503856871c001d closed
2015-10-05 16:31:57,574 INFO org.apache.zookeeper.ClientCnxn:
EventThread shut down
2015-10-05 16:31:57,577 INFO org.apache.hadoop.io.SequenceFile:
available bytes: 1
2015-10-05 16:31:57,581 INFO
org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
truncater with mapRetainSize=-1 and reduceRetainSize=-1
2015-10-05 16:31:57,583 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hdfs (auth:SIMPLE)
cause:java.io.IOException: keyvalues=NONE read 4096 bytes, should read
327381
2015-10-05 16:31:57,583 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: keyvalues=NONE read 4096 bytes, should read 327381
at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2221)
at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
2015-10-05 16:31:57,588 INFO org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
--
Håvard Wahl Kongsgård
Data Scientist
Re: Problem importing hbase exports
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
Just for the record the dumps/exports work with Cdh4, so a bug in
hadoop or hbase. So don't import older dumps to CDH5.
-Håvard
On Mon, Oct 5, 2015 at 4:54 PM, Håvard Wahl Kongsgård
<ha...@gmail.com> wrote:
> Hi, Iam having problems importing older hbase exports( hbase 0.94).
> Hadoop/hbase fails with "java.io.IOException: keyvalues=NONE read 4096
> bytes, should read 327381". What I have noticed is that the latest
> version of hbase import splits the files into multiple parts/task,
> while in hbase 0.9X import only created one job per file.
>
> My current system is 1.0.0-cdh5.4.7.
>
> command
> sudo -u hbase hbase -Dhbase.import.version=0.94
> org.apache.hadoop.hbase.mapreduce.Import photos
> hdfs://rack3:8020/hbase_photos
>
>
> Log
> 2015-10-05 16:31:55,005 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 2015-10-05 16:31:55,519 INFO
> org.apache.hadoop.conf.Configuration.deprecation: session.id is
> deprecated. Instead, use dfs.metrics.session-id
> 2015-10-05 16:31:55,520 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5-cdh5.0.0-beta-1--1, built on
> 10/28/2013 00:00 GMT
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:host.name=rack3
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.version=1.7.0_67
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.vendor=Oracle Corporation
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-7-oracle-cloudera/jre
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.class.path=/etc/hadoop/conf:/usr/lib/jvm/java-7-oracle-cloudera/lib/tools.jar:/usr/lib/hadoop-0.20-mapreduce:/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.7.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-2.1/jsp-api-2.1.jar:/etc/hadoop/conf:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/slf4j-log4j12.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.5.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/.//hadoop-nfs.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-common-2.2.0-cdh5.0.0-beta-1-tests.jar:/usr/lib/hadoop/.//hadoop-annotations-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-nfs-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-auth-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.2.0-cdh5.0.0-beta-1-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.7.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-examples.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-test.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-test-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-tools-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-examples-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-tools.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core-2.2.0-mr1-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/jars/classes:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/jars/job.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-912292882399157935_1505303099_946479403/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-common-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-6963687099100849172_377036872_946479477/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-1572551158342880427_72556552_946479527/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/htrace-core-2.01.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/2785619607140856733_609359336_946479607/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-protocol-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/5634015324466447495_1643919977_946479677/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-2466065489594964804_983196214_946479753/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hadoop-core-2.2.0-mr1-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/5926457346568904278_1528333591_946479800/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/protobuf-java-2.5.0.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/-9219457335404538790_2034210828_946479843/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-hadoop-compat-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/146573646801466628_-788370405_946479879/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-client-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/5567766410265772826_-1711486573_946480347/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/hbase-server-0.95.2-cdh5.0.0-beta-1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/4939585511868799146_854929102_946480411/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/guava-12.0.1.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/distcache/3116845128622493648_-856562511_946480494/0.0.0.0/var/lib/hadoop-hdfs/cache/mapred/mapred/staging/hdfs/.staging/job_201510051609_0001/libjars/netty-3.2.2.Final.jar:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/attempt_201510051609_0001_m_000000_2/work
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.library.path=/usr/lib/hadoop-0.20-mapreduce/lib/native/Linux-amd64-64:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/attempt_201510051609_0001_m_000000_2/work
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/attempt_201510051609_0001_m_000000_2/work/tmp
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2015-10-05 16:31:56,038 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.version=3.16.0-30-generic
> 2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.name=mapred
> 2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.home=/usr/lib/hadoop-0.20-mapreduce
> 2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.dir=/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/hdfs/jobcache/job_201510051609_0001/attempt_201510051609_0001_m_000000_2/work
> 2015-10-05 16:31:56,039 INFO org.apache.zookeeper.ZooKeeper:
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=90000 watcher=hconnection-0x1d63530a
> 2015-10-05 16:31:56,051 INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process
> identifier=hconnection-0x1d63530a connecting to ZooKeeper
> ensemble=localhost:2181
> 2015-10-05 16:31:56,053 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not
> attempt to authenticate using SASL (unknown error)
> 2015-10-05 16:31:56,054 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating
> session
> 2015-10-05 16:31:56,540 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server localhost/0:0:0:0:0:0:0:1:2181,
> sessionid = 0x1503856871c001b, negotiated timeout = 40000
> 2015-10-05 16:31:57,087 INFO
> org.apache.hadoop.hbase.mapreduce.TableOutputFormat: Created table
> instance for photos
> 2015-10-05 16:31:57,096 INFO org.apache.hadoop.util.ProcessTree:
> setsid exited with exit code 0
> 2015-10-05 16:31:57,100 INFO org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@404226a8
> 2015-10-05 16:31:57,416 INFO org.apache.hadoop.mapred.MapTask:
> Processing split:
> hdfs://0.0.0.0:8020/hbase_fb_photos_compressed/part-m-00033:2281701376+143727446
> 2015-10-05 16:31:57,456 INFO
> org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded &
> initialized native-zlib library
> 2015-10-05 16:31:57,457 INFO org.apache.hadoop.io.compress.CodecPool:
> Got brand-new decompressor [.deflate]
> 2015-10-05 16:31:57,548 INFO org.apache.zookeeper.ZooKeeper:
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=90000 watcher=attempt_201510051609_0001_m_000000_2
> 2015-10-05 16:31:57,549 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/127.0.0.1:2181. Will not attempt
> to authenticate using SASL (unknown error)
> 2015-10-05 16:31:57,550 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to localhost/127.0.0.1:2181, initiating session
> 2015-10-05 16:31:57,550 INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process
> identifier=attempt_201510051609_0001_m_000000_2 connecting to
> ZooKeeper ensemble=localhost:2181
> 2015-10-05 16:31:57,566 INFO org.apache.zookeeper.ClientCnxn: Session
> establishment complete on server localhost/127.0.0.1:2181, sessionid =
> 0x1503856871c001d, negotiated timeout = 40000
> 2015-10-05 16:31:57,574 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x1503856871c001d closed
> 2015-10-05 16:31:57,574 INFO org.apache.zookeeper.ClientCnxn:
> EventThread shut down
> 2015-10-05 16:31:57,577 INFO org.apache.hadoop.io.SequenceFile:
> available bytes: 1
> 2015-10-05 16:31:57,581 INFO
> org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
> truncater with mapRetainSize=-1 and reduceRetainSize=-1
> 2015-10-05 16:31:57,583 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hdfs (auth:SIMPLE)
> cause:java.io.IOException: keyvalues=NONE read 4096 bytes, should read
> 327381
> 2015-10-05 16:31:57,583 WARN org.apache.hadoop.mapred.Child: Error running child
> java.io.IOException: keyvalues=NONE read 4096 bytes, should read 327381
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2221)
> at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.nextKeyValue(SequenceFileRecordReader.java:74)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483)
> at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> 2015-10-05 16:31:57,588 INFO org.apache.hadoop.mapred.Task: Runnning
> cleanup for the task
>
>
> --
> Håvard Wahl Kongsgård
> Data Scientist
--
Håvard Wahl Kongsgård
Data Scientist