RE: Best practices for loading data into hbase

[David Poisson <Da...@ca.fujitsu.com>]

Hi,
     Thanks to everyone that replied!

@Tariq: Not childish at all. We don't have direct access to the database, but rather we will be going through a web service to obtain dumps of the data. This is why we are not using sqoop (unless sqoop would support such an operation, but not that I'm aware of).

@Ted: We're using CDH4:
rpm -q hbase
hbase-0.94.2+202-1.cdh4.2.0.p0.11.el6.x86_64
rpm -q hadoop
hadoop-2.0.0+922-1.cdh4.2.0.p0.12.el6.x86_64

@J-D: I've started researching how to set up pseudo-distributed mode on my vmware as per your suggestion, however I'm having a hard time connecting to my jobTracker. 
There's nothing in iptables, nothing in hosts.deny.
I can see I'm listening on that port, but I have a feeling I'm only listening on 127.0.0.1:8020 and not 0.0.0.0:8020:

[cloudera@locahost ~]$ netstat -lnt  | grep 8020
tcp        0      0 127.0.0.1:8020              0.0.0.0:*                   LISTEN

I can do the following: telnet 127.0.0.1 8020
However the following fails: telnet 10.9.2.194 8020

When I run my mapReduce job, here's the last few lines (for clarity):
13/06/05 10:33:50 INFO compress.CodecPool: Got brand-new compressor [.deflate]
13/06/05 10:33:50 INFO mapreduce.HFileOutputFormat: Incremental table output configured.
13/06/05 10:36:58 WARN conf.Configuration: session.id is deprecated. Instead, use dfs.metrics.session-id
13/06/05 10:36:58 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
13/06/05 10:37:16 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/06/05 10:37:16 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/06/05 10:37:30 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-cloudera/mapred/staging/cloudera283903860/.staging/job_local283903860_0001
13/06/05 10:37:30 ERROR security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 10.9.2.194:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
13/06/05 10:37:30 ERROR sourcestaging.ReducerXML: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 10.9.2.194:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

ssh keys have been generated so I can ssh from my vmware into my vmware with user cloudera without passwords.

The following are set in the configs (as per http://hadoop.apache.org/docs/r1.1.1/single_node_setup.html#PseudoDistributed):
conf/core-site.xml:

<configuration>
     <property>
         <name>fs.defaultFS</name> ** I changed fs.default.name to fs.defaultFS, I tried them both and neither allowed me to connect
         <value>hdfs://localhost:9000</value>
     </property>
</configuration>


conf/hdfs-site.xml:

<configuration>
     <property>
         <name>dfs.replication</name>
         <value>1</value>
     </property>
</configuration>


conf/mapred-site.xml:

<configuration>
     <property>
         <name>mapred.job.tracker</name>
         <value>localhost:9001</value>
     </property>
</configuration>


Here's my /etc/hosts file:
[cloudera@localhost /etc/alternatives/hadoop-conf]$ cat /etc/hosts
127.0.0.1		localhost.localdomain localhost
::1		localhost6.localdomain6 localhost6

We have a pretty strict proxy here, could it be interfering? Other than that, my VM's networking is set to bridged, if that makes any difference. Mind you, I'm trying to connect from my vm to my vm.

I'm at a lost here. Could really use some guidance. Thanks!

David

________________________________________
From: David Poisson [David.Poisson@ca.fujitsu.com]
Sent: Friday, May 31, 2013 4:19 PM
To: user@hbase.apache.org
Subject: Best practices for loading data into hbase

Hi,
     We are still very new at all of this hbase/hadoop/mapreduce stuff. We are looking for the best practices that will fit our requirements. We are currently using the latest cloudera vmware's (single node) for our development tests.

The problem is as follows:

We have multiple sources in different format (xml, csv, etc), which are dumps of existing systems. As one might think, there will be an initial "import" of the data into hbase
and afterwards, the systems would most likely dump whatever data they have accumulated since the initial import into hbase or since the last data dump. Another thing, we would require to have an
intermediary step, so that we can ensure all of a source's data can be successfully processed, something which would look like:

XML data file --(MR JOB)--> Intermediate (hbase table or hfile?) --(MR JOB)--> production tables in hbase

We're guessing we can't use something like a transaction in hbase, so we thought about using a intermediate step: Is that how things are normally done?

As we import data into hbase, we will be populating several tables that links data parts together (account X in System 1 == account Y in System 2) as tuples in 3 tables. Currently,
this is being done by a mapreduce job which reads the XML source and uses multiTableOutputFormat to "put" data into those 3 hbase tables. This method
isn't that fast using our test sample (2 minutes for 5Mb), so we are looking at optimizing the loading of data.

We have been researching bulk loading but we are unsure of a couple of things:
Once we process an xml file and we populate our 3 "production" hbase tables, could we bulk load another xml file and append this new data to our 3 tables or would it write over what was written before?
In order to bulk load, we need to output a file using HFileOutputFormat. Since MultiHFileOutputFormat doesn't seem to officially exist yet (still in the works, right?), should we process our input xml file
with 3 MapReduce jobs instead of 1 and output an hfile for each, which we could then become our intermediate step (if all 3 hfiles were created without errors, then process was successful: bulk load
in hbase)? Can you experiment with bulk loading on a vmware? We're experiencing problems with partition file not being found with the following exception:

java.lang.Exception: java.lang.IllegalArgumentException: Can't read partitions file
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:404)
Caused by: java.lang.IllegalArgumentException: Can't read partitions file
        at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:108)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:588)

We also tried another idea on how to speed things up: What if instead of doing individual puts, we passed a list of puts to put() (eg: htable.put(putList) ). Internally in hbase, would there be less overhead vs multiple
calls to put()? It seems to be faster, however since we're not using context.write, I'm guessing this will lead to problems later on, right?

Turning off WAL on puts to speed things up isn't an option, since data loss would be unacceptable, even if the chances of a failure occurring are slim.

Thanks, David