You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "sydt (JIRA)" <ji...@apache.org> on 2017/02/22 04:18:44 UTC
[jira] [Comment Edited] (SPARK-19661) Spark-2.1.0 can not connect
hbase
[ https://issues.apache.org/jira/browse/SPARK-19661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877441#comment-15877441 ]
sydt edited comment on SPARK-19661 at 2/22/17 4:18 AM:
-------------------------------------------------------
Yeah,you are right.It is accomplished by hive-hbase-handler-0.13.1.jar actually. In hive, we can create table by
SET hbase.zookeeper.quorum=zkNode1,zkNode2,zkNode3;
SET zookeeper.znode.parent=/hbase;
ADD jar /usr/local/apache-hive-0.13.1-bin/lib/hive-hbase-handler-0.13.1.jar;
CREATE EXTERNAL TABLE lxw1234 (
rowkey string,
f1 map<STRING,STRING>,
f2 map<STRING,STRING>,
f3 map<STRING,STRING>
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,f1:,f2:,f3:")
TBLPROPERTIES ("hbase.table.name" = "lxw1234");
After
INSERT INTO TABLE lxw1234
SELECT 'row1' AS rowkey,
map('c3','name3') AS f1,
map('c3','age3') AS f2,
map('c4','job3') AS f3
FROM DUAL
limit 1;
now, hive can access table lxw1234 like ordinary external table and it it stored in hbase. Sparksql has the same syntax parser as hive and it should support this SQL syntax in spark-1.6.2.However, in spark-2.1.0,it does not.
was (Author: wangchao2017):
Yeah,you are right.It is accomplished by hive-hbase-handler-0.13.1.jar actually. In hive, we can create table by
SET hbase.zookeeper.quorum=zkNode1,zkNode2,zkNode3;
SET zookeeper.znode.parent=/hbase;
ADD jar /usr/local/apache-hive-0.13.1-bin/lib/hive-hbase-handler-0.13.1.jar;
CREATE EXTERNAL TABLE lxw1234 (
rowkey string,
f1 map<STRING,STRING>,
f2 map<STRING,STRING>,
f3 map<STRING,STRING>
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,f1:,f2:,f3:")
TBLPROPERTIES ("hbase.table.name" = "lxw1234");
After
INSERT INTO TABLE lxw1234
SELECT 'row1' AS rowkey,
map('c3','name3') AS f1,
map('c3','age3') AS f2,
map('c4','job3') AS f3
FROM DUAL
limit 1;
now, hive can access table lxw1234 like ordinary external table and it it stored in hbase.
> Spark-2.1.0 can not connect hbase
> ---------------------------------
>
> Key: SPARK-19661
> URL: https://issues.apache.org/jira/browse/SPARK-19661
> Project: Spark
> Issue Type: Bug
> Components: Deploy
> Affects Versions: 2.1.0
> Reporter: sydt
>
> When spark-sql of spark-2.1.0 connect hbase by
> CREATE EXTERNAL TABLE lxw123(
> rowkey string,
> f1 map<STRING,STRING>,
> f2 map<STRING,STRING>,
> f3 map<STRING,STRING>
> ) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,f1:,f2:,f3:")
> TBLPROPERTIES ("hbase.table.name" = "lxw1234");
> it has no response and show
> Error in query:
> Operation not allowed: STORED BY(line 6, pos 2)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org