Hadoop pipes with Hbase as input

[Håvard Wahl Kongsgård <ha...@gmail.com>]

Hi, when I attempt to use a hbase table as input hadoop just seems to
stall. I don't get any errors.

I can't find any examples where pipes are used with hbase, is it
possible at all? Streaming seems to be an alternative
http://dumbotics.com/2009/07/31/dumbo-over-hbase/

hadoop pipes -conf myconf_job.conf -input name_of_table -output /tmp/out

12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.5-cdh3u4--1, built on 05/07/2012
21:08 GMT
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:host.name=kongs1
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_31
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.31/jre
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/usr/lib/hadoop-0.20/conf:/usr/lib/jvm/java-6-sun//lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20/lib/commons-lang-2.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/guava-r09-jarjar.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u4.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.cloudera.1.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hbase/hbase-0.90.6-cdh3u4.jar:/usr/lib/zookeeper/zookeeper-3.3.5-cdh3u4.jar
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-41-server
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:user.home=/usr/lib/hadoop-0.20
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/home/havard/d/graph
12/08/15 11:27:54 INFO zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=180000
watcher=hconnection
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Opening socket connection
to server localhost/127.0.0.1:2181
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Socket connection
established to localhost/127.0.0.1:2181, initiating session
12/08/15 11:27:54 INFO zookeeper.ClientCnxn: Session establishment
complete on server localhost/127.0.0.1:2181, sessionid =
0x139266be8b90004, negotiated timeout = 40000

my job conf

<property>
<name>mapred.input.format.class</name>
<value>org.apache.hadoop.hbase.mapred.TableInputFormat</value>
</property>

<property>
  <name>hadoop.pipes.java.recordreader</name>
  <value>true</value>
</property>

<property>
<name>hbase.mapred.tablecolumns</name>
<value>col_fam:name</value>
</property>


-Håvard