You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by "Savant, Keshav" <Ke...@fisglobal.com> on 2012/02/28 13:28:43 UTC

Hive | HBase Integration

Hi All,

We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1.

In our next step we are eyeing HBase integration with Hive. As far as we understand from articles available on internet and apache site, we can use HBase instead of derby as a metastore of Hive, this gives us more flexibility while handling very large data.

We are using hbase-0.92.0 to integrate it with Hive, till now HBase has been setup and we can create sample table on it and insert sample data in it, but we are not able to integrate it with Hive, because when we issue the command to create hive specific table on HBase (below in box) the command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.

CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek");

Please provide us some pointers (steps to follow) for doing this integration or what we are not doing correctly. Till now we got these below URLs to do this, any help is appreciated

http://mevivs.wordpress.com/2010/11/24/hivehbase-integration/
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration

Kind regards,
Keshav C Savant

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: Hive | HBase Integration

Posted by Bejoy Ks <be...@yahoo.com>.

Hi Rinku
      For insert into hbase, it'd be better you  depend on Hbase APIs for the same. I'm not sure on the APIS in hbase that would support bulk loading, you could find better solution on the hbase mailing list.
     There is a way to insert data from hive into hbase . You need to CREATE a hive managed TABLE in hive that uses a Hbase storage handler. The below url can help you with the details.

https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration#HBaseIntegration-Usage

 From the execption you posted , hive is not able to connect to hbase master and make the curresponding table creation in hbase. Is your hbase installation good? have to tried creating tables and loading data on to them?


It could be some jar mismatches as well. Follow the first part of the url,basically in building your Storage Handler (hive-hbase) .

$HIVE_SRC/build/dist/bin/hive --auxpath $HIVE_SRC/build/dist/lib/hive-hbase-handler-0.7.0.jar,$HIVE_SRC/build/dist/lib/hbase-0.89.0-SNAPSHOT.jar,$HIVE_SRC/build/dist/lib/zookeeper-3.3.1.jar -hiveconf hbase.zookeeper.quorum=zk1.yoyodyne.com,zk2.yoyodyne.com,zk3.yoyodyne.com

Regards
Bejoy.K.S


________________________________
 From: "Garg, Rinku" <Ri...@fisglobal.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; Bejoy Ks <be...@yahoo.com> 
Sent: Thursday, March 1, 2012 11:19 AM
Subject: RE: Hive | HBase Integration
 

 
Hi Bejoy,
 
We want to insert data into Hbase cluster. We are having data in CSV files. We want to insert large number of CSV’s files into hbase and  then quering the hbase cluster.
 
Please guide us on how to proceed.
 
We tried to create the table in HIVE using following command.
CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping"= ":key,cf1:val") TBLPROPERTIES ("hbase.table.name"= "xyz");
 
But we are getting the following exception:
 
FAILED: Error in metadata: java.lang.IllegalArgumentException: Not a host:port pair: ï¿½
26156@mastermaster,60000,1330393836186
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
 
Please help.
 
Thanks
Rinku Garg
 
From:Bejoy Ks [mailto:bejoy_ks@yahoo.com] 
Sent: 29 February 2012 17:51
To: user@hive.apache.org
Subject: Re: Hive | HBase Integration
 
Hi Rinku 
 
We want to use hadoop’s hdfs with hbase cluster
Can you elaborate more on this? 
 
·        Do we still need hive? 
You don't need Hive for Hbase to work. Hbase doesn't support SQL like queries, to support SQL like querying in Hbase we create a hive table that maps to the hbase table. The data is stored only on the hbase Region server and not brought to hdfs. Hbase and hive are primarily meant for two different purposes
Hive - is a DWH system on top of hadoop, intended for batch processing. It supports SQL like commads and is mostly used for Analytic purposes .
Hbase - is a low latency noSQL,distributed database in hadoop eco system. It is meant for point to point queries, frequent inserts, updates etc.
 
 
·        Can we directly use hbase with hadoop.
Definitely, just set up a hadoop cluster and then install hbase master and region servers on the same. Are you looking st some specific operation here?
 
·        If we have to use hbase through hive then how can we proceed?
AFAIK, the hive Hbase integration is meant to query data from Hbase alone. For inserts and updates in Hbase tables you need to use the Hbase APIs itself. 
 
 
Regards
Bejoy.K.S
 
 

________________________________
 
From:"Garg, Rinku" <Ri...@fisglobal.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; "bejoy_ks@yahoo.com" <be...@yahoo.com> 
Sent: Wednesday, February 29, 2012 11:19 AM
Subject: RE: Hive | HBase Integration


Hi Bejoy,
 
Thanks for the quick reply.
 
Yes, we can create normal table in Hive, but here we are trying to create table in hbase to integrate it with HIVE.
 
Here is the scenario.
 
We want to use hadoop’s hdfs with hbase cluster. Earlier we have successful setup of hadoop-0.20.203.0 and hive-0.7.1. We can insert our csv’s into HDFS using hive and query it. But now we want to use HBASE cluster. So our queries are 
 
·        Do we still need hive?  
·        Can we directly use hbase with hadoop.
·        If we have to use hbase through hive then how can we proceed?
 
Please help.
 
Thanks
Rinku Garg
 
From:bejoy_ks@yahoo.com [mailto:bejoy_ks@yahoo.com] 
Sent: 29 February 2012 11:08
To: user@hive.apache.org
Subject: Re: Hive | HBase Integration
 
Hi Rinku
Were you able to create a normal table within your hive without any issues? By Normal table I mean the one that has data dir in hdfs not in HBase. 
Regards
Bejoy K S

From handheld, Please excuse typos.

________________________________
 
From:  "Garg, Rinku" <Ri...@fisglobal.com> 
Date:  Wed, 29 Feb 2012 05:29:12 +0000
To:  user@hive.apache.org<us...@hive.apache.org>; Bejoy Ks<be...@yahoo.com>
ReplyTo: user@hive.apache.org 
Subject: RE: Hive | HBase Integration
 
Hi ,
 
We tried the same also by issuing the below mentioned command but command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.
 
CREATE EXTERNAL TABLE hive_hbasetable_k(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "hivehbasek");    
 
Please suggest.
 
Thanks
Rinku Garg
 
From:Bejoy Ks [mailto:bejoy_ks@yahoo.com] 
Sent: 29 February 2012 10:33
To: user@hive.apache.org
Subject: Re: Hive | HBase Integration
 
Hi Keshav
      Make your hive table EXTERNAL, it should get things rolling. If you are mapping hive to a Hbase table, then as a mandatory requirement the hive table has to be EXTERNAL.
 
Hope it helps.
 
Regards
Bejoy.K.S
 

________________________________
 
From:"Savant, Keshav" <Ke...@fisglobal.com>
To: "user@hive.apache.org" <us...@hive.apache.org> 
Sent: Tuesday, February 28, 2012 5:58 PM
Subject: Hive | HBase Integration
 
Hi All,
 
We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. 
 
In our next step we are eyeing HBase integration with Hive. As far as we understand from articles available on internet and apache site, we can use HBase instead of derby as a metastore of Hive, this gives us more flexibility while handling very large data.
 
We are using hbase-0.92.0 to integrate it with Hive, till now HBase has been setup and we can create sample table on it and insert sample data in it, but we are not able to integrate it with Hive, because when we issue the command to create hive specific table on HBase (below in box) the command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.
 
CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek"); 
 
 
Please provide us some pointers (steps to follow) for doing this integration or what we are not doing correctly. Till now we got these below URLs to do this, any help is appreciated
 
http://mevivs.wordpress.com/2010/11/24/hivehbase-integration/
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration
 
Kind regards,
Keshav C Savant 
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender
 immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
 
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender
 immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender
 immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
 
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

RE: Hive | HBase Integration

Posted by "Garg, Rinku" <Ri...@fisglobal.com>.

Hi Bejoy,

We want to insert data into Hbase cluster. We are having data in CSV files. We want to insert large number of CSV’s files into hbase and  then quering the hbase cluster.

Please guide us on how to proceed.

We tried to create the table in HIVE using following command.
CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz");

But we are getting the following exception:

FAILED: Error in metadata: java.lang.IllegalArgumentException: Not a host:port pair: ï¿½
26156@mastermaster,60000,1330393836186
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

Please help.

Thanks
Rinku Garg

From: Bejoy Ks [mailto:bejoy_ks@yahoo.com]
Sent: 29 February 2012 17:51
To: user@hive.apache.org
Subject: Re: Hive | HBase Integration

Hi Rinku

We want to use hadoop’s hdfs with hbase cluster
Can you elaborate more on this?

•         Do we still need hive?
You don't need Hive for Hbase to work. Hbase doesn't support SQL like queries, to support SQL like querying in Hbase we create a hive table that maps to the hbase table. The data is stored only on the hbase Region server and not brought to hdfs. Hbase and hive are primarily meant for two different purposes
Hive - is a DWH system on top of hadoop, intended for batch processing. It supports SQL like commads and is mostly used for Analytic purposes .
Hbase - is a low latency noSQL,distributed database in hadoop eco system. It is meant for point to point queries, frequent inserts, updates etc.


•         Can we directly use hbase with hadoop.
Definitely, just set up a hadoop cluster and then install hbase master and region servers on the same. Are you looking st some specific operation here?

•         If we have to use hbase through hive then how can we proceed?
AFAIK, the hive Hbase integration is meant to query data from Hbase alone. For inserts and updates in Hbase tables you need to use the Hbase APIs itself.


Regards
Bejoy.K.S


________________________________
From: "Garg, Rinku" <Ri...@fisglobal.com>>
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>; "bejoy_ks@yahoo.com<ma...@yahoo.com>" <be...@yahoo.com>>
Sent: Wednesday, February 29, 2012 11:19 AM
Subject: RE: Hive | HBase Integration


Hi Bejoy,

Thanks for the quick reply.

Yes, we can create normal table in Hive, but here we are trying to create table in hbase to integrate it with HIVE.

Here is the scenario.

We want to use hadoop’s hdfs with hbase cluster. Earlier we have successful setup of hadoop-0.20.203.0 and hive-0.7.1. We can insert our csv’s into HDFS using hive and query it. But now we want to use HBASE cluster. So our queries are

•         Do we still need hive?
•         Can we directly use hbase with hadoop.
•         If we have to use hbase through hive then how can we proceed?

Please help.

Thanks
Rinku Garg

From: bejoy_ks@yahoo.com<ma...@yahoo.com> [mailto:bejoy_ks@yahoo.com]
Sent: 29 February 2012 11:08
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Hive | HBase Integration

Hi Rinku
Were you able to create a normal table within your hive without any issues? By Normal table I mean the one that has data dir in hdfs not in HBase.
Regards
Bejoy K S

From handheld, Please excuse typos.
________________________________
From: "Garg, Rinku" <Ri...@fisglobal.com>>
Date: Wed, 29 Feb 2012 05:29:12 +0000
To: user@hive.apache.org<us...@hive.apache.org>>; Bejoy Ks<be...@yahoo.com>>
ReplyTo: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Hive | HBase Integration

Hi ,

We tried the same also by issuing the below mentioned command but command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.

CREATE EXTERNAL TABLE hive_hbasetable_k(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "hivehbasek");

Please suggest.

Thanks
Rinku Garg

From: Bejoy Ks [mailto:bejoy_ks@yahoo.com]
Sent: 29 February 2012 10:33
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Hive | HBase Integration

Hi Keshav
      Make your hive table EXTERNAL, it should get things rolling. If you are mapping hive to a Hbase table, then as a mandatory requirement the hive table has to be EXTERNAL.

Hope it helps.

Regards
Bejoy.K.S

________________________________
From: "Savant, Keshav" <Ke...@fisglobal.com>>
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Sent: Tuesday, February 28, 2012 5:58 PM
Subject: Hive | HBase Integration

Hi All,

We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1.

In our next step we are eyeing HBase integration with Hive. As far as we understand from articles available on internet and apache site, we can use HBase instead of derby as a metastore of Hive, this gives us more flexibility while handling very large data.

We are using hbase-0.92.0 to integrate it with Hive, till now HBase has been setup and we can create sample table on it and insert sample data in it, but we are not able to integrate it with Hive, because when we issue the command to create hive specific table on HBase (below in box) the command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.

CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek");



Please provide us some pointers (steps to follow) for doing this integration or what we are not doing correctly. Till now we got these below URLs to do this, any help is appreciated

http://mevivs.wordpress.com/2010/11/24/hivehbase-integration/
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration

Kind regards,
Keshav C Savant
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.


_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: Hive | HBase Integration

Posted by Bejoy Ks <be...@yahoo.com>.

Hi Rinku 


We want to use hadoop’s hdfs with hbase cluster
Can you elaborate more on this? 


·         Do we still need hive? 

You don't need Hive for Hbase 
to work. Hbase doesn't support SQL like queries, to support SQL like 
querying in Hbase we create a hive table that maps to the hbase table. The data is stored only on the hbase Region server and not brought to hdfs. Hbase and hive are primarily meant for two different purposes
Hive - is a DWH system on top of hadoop, intended for batch processing. It supports SQL like commads and is mostly used for Analytic purposes .

Hbase - is a low latency noSQL,distributed database in hadoop eco system. It is meant for point to point queries, frequent inserts, updates etc.



·         Can we directly use hbase with hadoop.
Definitely, just set up a hadoop cluster and then install hbase master and region servers on the same. Are you looking st some specific operation here?


·         If we have to use hbase through hive then how can we proceed?
AFAIK, the hive Hbase integration is meant to query data from Hbase alone. For inserts and updates in Hbase tables you need to use the Hbase APIs itself. 



Regards
Bejoy.K.S




________________________________
 From: "Garg, Rinku" <Ri...@fisglobal.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; "bejoy_ks@yahoo.com" <be...@yahoo.com> 
Sent: Wednesday, February 29, 2012 11:19 AM
Subject: RE: Hive | HBase Integration
 

 
Hi Bejoy,
 
Thanks for the quick reply.
 
Yes, we can create normal table in Hive, but here we are trying to create table in hbase to integrate it with HIVE.
 
Here is the scenario.
 
We want to use hadoop’s hdfs with hbase cluster. Earlier we have successful setup of hadoop-0.20.203.0 and hive-0.7.1. We can insert our csv’s into HDFS using hive and query it. But now we want to use HBASE cluster. So our queries are 
 
·         Do we still need hive? 
·         Can we directly use hbase with hadoop.
·         If we have to use hbase through hive then how can we proceed?
 
Please help.
 
Thanks
Rinku Garg
 
From:bejoy_ks@yahoo.com [mailto:bejoy_ks@yahoo.com] 
Sent: 29 February 2012 11:08
To: user@hive.apache.org
Subject: Re: Hive | HBase Integration
 
Hi Rinku
Were you able to create a normal table within your hive without any issues? By Normal table I mean the one that has data dir in hdfs not in HBase. 
Regards
Bejoy K S

From handheld, Please excuse typos.

________________________________
 
From: "Garg, Rinku" <Ri...@fisglobal.com> 
Date: Wed, 29 Feb 2012 05:29:12 +0000
To: user@hive.apache.org<us...@hive.apache.org>; Bejoy Ks<be...@yahoo.com>
ReplyTo: user@hive.apache.org 
Subject: RE: Hive | HBase Integration
 
Hi ,
 
We tried the same also by issuing the below mentioned command but command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.
 
CREATE EXTERNAL TABLE hive_hbasetable_k(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "hivehbasek");    
 
Please suggest.
 
Thanks
Rinku Garg
 
From:Bejoy Ks [mailto:bejoy_ks@yahoo.com] 
Sent: 29 February 2012 10:33
To: user@hive.apache.org
Subject: Re: Hive | HBase Integration
 
Hi Keshav
      Make your hive table EXTERNAL, it should get things rolling. If you are mapping hive to a Hbase table, then as a mandatory requirement the hive table has to be EXTERNAL.
 
Hope it helps.
 
Regards
Bejoy.K.S
 

________________________________
 
From:"Savant, Keshav" <Ke...@fisglobal.com>
To: "user@hive.apache.org" <us...@hive.apache.org> 
Sent: Tuesday, February 28, 2012 5:58 PM
Subject: Hive | HBase Integration
 
Hi All,
 
We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. 
 
In our next step we are eyeing HBase integration with Hive. As far as we understand from articles available on internet and apache site, we can use HBase instead of derby as a metastore of Hive, this gives us more flexibility while handling very large data.
 
We are using hbase-0.92.0 to integrate it with Hive, till now HBase has been setup and we can create sample table on it and insert sample data in it, but we are not able to integrate it with Hive, because when we issue the command to create hive specific table on HBase (below in box) the command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.
 
CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek"); 
 
 
Please provide us some pointers (steps to follow) for doing this integration or what we are not doing correctly. Till now we got these below URLs to do this, any help is appreciated
 
http://mevivs.wordpress.com/2010/11/24/hivehbase-integration/
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration
 
Kind regards,
Keshav C Savant 
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender
 immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
 
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender
 immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

RE: Hive | HBase Integration

Posted by "Garg, Rinku" <Ri...@fisglobal.com>.

Hi Bejoy,

Thanks for the quick reply.

Yes, we can create normal table in Hive, but here we are trying to create table in hbase to integrate it with HIVE.

Here is the scenario.

We want to use hadoop's hdfs with hbase cluster. Earlier we have successful setup of hadoop-0.20.203.0 and hive-0.7.1. We can insert our csv's into HDFS using hive and query it. But now we want to use HBASE cluster. So our queries are


*         Do we still need hive?

*         Can we directly use hbase with hadoop.

*         If we have to use hbase through hive then how can we proceed?

Please help.

Thanks
Rinku Garg

From: bejoy_ks@yahoo.com [mailto:bejoy_ks@yahoo.com]
Sent: 29 February 2012 11:08
To: user@hive.apache.org
Subject: Re: Hive | HBase Integration

Hi Rinku
Were you able to create a normal table within your hive without any issues? By Normal table I mean the one that has data dir in hdfs not in HBase.
Regards
Bejoy K S

>From handheld, Please excuse typos.
________________________________
From: "Garg, Rinku" <Ri...@fisglobal.com>>
Date: Wed, 29 Feb 2012 05:29:12 +0000
To: user@hive.apache.org<us...@hive.apache.org>>; Bejoy Ks<be...@yahoo.com>>
ReplyTo: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Hive | HBase Integration

Hi ,

We tried the same also by issuing the below mentioned command but command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.

CREATE EXTERNAL TABLE hive_hbasetable_k(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "hivehbasek");

Please suggest.

Thanks
Rinku Garg

From: Bejoy Ks [mailto:bejoy_ks@yahoo.com]
Sent: 29 February 2012 10:33
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Hive | HBase Integration

Hi Keshav
      Make your hive table EXTERNAL, it should get things rolling. If you are mapping hive to a Hbase table, then as a mandatory requirement the hive table has to be EXTERNAL.

Hope it helps.

Regards
Bejoy.K.S

________________________________
From: "Savant, Keshav" <Ke...@fisglobal.com>>
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Sent: Tuesday, February 28, 2012 5:58 PM
Subject: Hive | HBase Integration

Hi All,

We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1.

In our next step we are eyeing HBase integration with Hive. As far as we understand from articles available on internet and apache site, we can use HBase instead of derby as a metastore of Hive, this gives us more flexibility while handling very large data.

We are using hbase-0.92.0 to integrate it with Hive, till now HBase has been setup and we can create sample table on it and insert sample data in it, but we are not able to integrate it with Hive, because when we issue the command to create hive specific table on HBase (below in box) the command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.

CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek");



Please provide us some pointers (steps to follow) for doing this integration or what we are not doing correctly. Till now we got these below URLs to do this, any help is appreciated

http://mevivs.wordpress.com/2010/11/24/hivehbase-integration/
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration

Kind regards,
Keshav C Savant
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: Hive | HBase Integration

Posted by be...@yahoo.com.

Hi Rinku
       Were you able to create a normal table within your hive without any issues? By Normal table I mean the one that has data dir in hdfs not in HBase. 

Regards
Bejoy K S

From handheld, Please excuse typos.

-----Original Message-----
From: "Garg, Rinku" <Ri...@fisglobal.com>
Date: Wed, 29 Feb 2012 05:29:12 
To: user@hive.apache.org<us...@hive.apache.org>; Bejoy Ks<be...@yahoo.com>
Reply-To: user@hive.apache.org
Subject: RE: Hive | HBase Integration

Hi ,

We tried the same also by issuing the below mentioned command but command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.

CREATE EXTERNAL TABLE hive_hbasetable_k(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "hivehbasek");

Please suggest.

Thanks
Rinku Garg

From: Bejoy Ks [mailto:bejoy_ks@yahoo.com]
Sent: 29 February 2012 10:33
To: user@hive.apache.org
Subject: Re: Hive | HBase Integration

Hi Keshav
      Make your hive table EXTERNAL, it should get things rolling. If you are mapping hive to a Hbase table, then as a mandatory requirement the hive table has to be EXTERNAL.

Hope it helps.

Regards
Bejoy.K.S

________________________________
From: "Savant, Keshav" <Ke...@fisglobal.com>>
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Sent: Tuesday, February 28, 2012 5:58 PM
Subject: Hive | HBase Integration

Hi All,

We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1.

In our next step we are eyeing HBase integration with Hive. As far as we understand from articles available on internet and apache site, we can use HBase instead of derby as a metastore of Hive, this gives us more flexibility while handling very large data.

We are using hbase-0.92.0 to integrate it with Hive, till now HBase has been setup and we can create sample table on it and insert sample data in it, but we are not able to integrate it with Hive, because when we issue the command to create hive specific table on HBase (below in box) the command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.

CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek");



Please provide us some pointers (steps to follow) for doing this integration or what we are not doing correctly. Till now we got these below URLs to do this, any help is appreciated

http://mevivs.wordpress.com/2010/11/24/hivehbase-integration/
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration

Kind regards,
Keshav C Savant
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

RE: Hive | HBase Integration

Posted by "Garg, Rinku" <Ri...@fisglobal.com>.

Hi ,

We tried the same also by issuing the below mentioned command but command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.

CREATE EXTERNAL TABLE hive_hbasetable_k(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "hivehbasek");

Please suggest.

Thanks
Rinku Garg

From: Bejoy Ks [mailto:bejoy_ks@yahoo.com]
Sent: 29 February 2012 10:33
To: user@hive.apache.org
Subject: Re: Hive | HBase Integration

Hi Keshav
      Make your hive table EXTERNAL, it should get things rolling. If you are mapping hive to a Hbase table, then as a mandatory requirement the hive table has to be EXTERNAL.

Hope it helps.

Regards
Bejoy.K.S

________________________________
From: "Savant, Keshav" <Ke...@fisglobal.com>>
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Sent: Tuesday, February 28, 2012 5:58 PM
Subject: Hive | HBase Integration

Hi All,

We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1.

In our next step we are eyeing HBase integration with Hive. As far as we understand from articles available on internet and apache site, we can use HBase instead of derby as a metastore of Hive, this gives us more flexibility while handling very large data.

We are using hbase-0.92.0 to integrate it with Hive, till now HBase has been setup and we can create sample table on it and insert sample data in it, but we are not able to integrate it with Hive, because when we issue the command to create hive specific table on HBase (below in box) the command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.

CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek");



Please provide us some pointers (steps to follow) for doing this integration or what we are not doing correctly. Till now we got these below URLs to do this, any help is appreciated

http://mevivs.wordpress.com/2010/11/24/hivehbase-integration/
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration

Kind regards,
Keshav C Savant
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: Hive | HBase Integration

Posted by Bejoy Ks <be...@yahoo.com>.

Hi Keshav
      Make your hive table EXTERNAL, it should get things rolling. If you are mapping hive to a Hbase table, then as a mandatory requirement the hive table has to be EXTERNAL.

Hope it helps.


Regards
Bejoy.K.S



________________________________
 From: "Savant, Keshav" <Ke...@fisglobal.com>
To: "user@hive.apache.org" <us...@hive.apache.org> 
Sent: Tuesday, February 28, 2012 5:58 PM
Subject: Hive | HBase Integration
 

 
Hi All,
 
We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. 
 
In our next step we are eyeing HBase integration with Hive. As far as we understand from articles available on internet and apache site, we can use HBase instead of derby as a metastore of Hive, this gives us more flexibility while handling very large data.
 
We are using hbase-0.92.0 to integrate it with Hive, till now HBase has been setup and we can create sample table on it and insert sample data in it, but we are not able to integrate it with Hive, because when we issue the command to create hive specific table on HBase (below in box) the command does not executes completely and a new command line is shown with an asterisk (*), and table does not gets created.
 
CREATE TABLE hive_hbasetable_k(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "hivehbasek"); 
 
 
Please provide us some pointers (steps to follow) for doing this integration or what we are not doing correctly. Till now we got these below URLs to do this, any help is appreciated
 
http://mevivs.wordpress.com/2010/11/24/hivehbase-integration/
https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration
 
Kind regards,
Keshav C Savant 
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.