You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hao Ren <h....@claravista.fr> on 2013/08/01 16:45:16 UTC
Why HBase integation with Hive makes Hive slow
Hi,
I have a cluster (1 master + 3 slaves) on which there Hive, Hbase, and
Hadoop.
In order to do some daily row-level update routine, we need to integrate
Hbase with hive, but the performance is not good.
E.g. There are 2 tables in hive,
hbase_table: a hbase table created via Hive
hive_table: a native hive table
both hold the same data set.
When runing:
select count(*) from hbase_table; ===> takes 500 s
select count(*) from hive_table; ===> takes 6 s
I have tried a lot of queries on the two tables. But hbase_table is
always very slow.
To be claire, I created the hbase_ table as below:
CREATE TABLE hbase_table (
idvisite string,
client_list Array<string>,
nb_client int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" =
":key,clients:id_list,clients:nb")
TBLPROPERTIES("hbase.table.name" = "table_test")
;
And my Hbase is on pseudo-distributed mode.
I guess, at the beginning of a hive query execution, hive will load data
from Hbase, where serde takes a long time.
Could someone tell me how to improve my poor performance ?
Is this cause by my wrongly configured integration ?
Is a fully-distributed mode needed here ?
Thank you in advance for your time.
Hao.
--
Hao Ren
ClaraVista
www.claravista.fr
Re: Why HBase integation with Hive makes Hive slow
Posted by Andrew Purtell <ap...@apache.org>.
This is a NPE from Hive code.
But it looks like you are running Hive on Shark?
So this looks like an inquiry for the Shark mailing list.
Good luck!
On Fri, Aug 2, 2013 at 8:04 AM, Hao Ren <h....@claravista.fr> wrote:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.serde2.**objectinspector.primitive.**
> WritableIntObjectInspector.**get(**WritableIntObjectInspector.**java:35)
> at org.apache.hadoop.hive.serde2.**lazy.LazyUtils.**
> writePrimitiveUTF8(LazyUtils.**java:199)
> at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:696)
> at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:758)
> at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:713)
> at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:758)
> at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:713)
> at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:685)
> at org.apache.hadoop.hive.hbase.**HBaseSerDe.serializeField(**
> HBaseSerDe.java:648)
> at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:560)
> at org.apache.hadoop.hive.ql.**exec.FileSinkOperator.**
> processOp(FileSinkOperator.**java:568)
> at shark.execution.**FileSinkOperator$anonfun$**
> processPartition$1.apply(**FileSinkOperator.scala:73)
> at shark.execution.**FileSinkOperator$anonfun$**
> processPartition$1.apply(**FileSinkOperator.scala:72)
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Why HBase integation with Hive makes Hive slow
Posted by Hao Ren <h....@claravista.fr>.
Thank you, lars
The performance is largely improved when setting scanner caching to 10000
But I still encounter a problem.
When loading data to a hbast table via hive, I got a NullPointrExecption:
java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:35)
at
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:199)
at
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:696)
at
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:758)
at
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:713)
at
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:758)
at
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:713)
at
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:685)
at
org.apache.hadoop.hive.hbase.HBaseSerDe.serializeField(HBaseSerDe.java:648)
at
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:560)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:568)
at
shark.execution.FileSinkOperator$$anonfun$processPartition$1.apply(FileSinkOperator.scala:73)
at
shark.execution.FileSinkOperator$$anonfun$processPartition$1.apply(FileSinkOperator.scala:72)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.Iterator$$anon$19.foreach(Iterator.scala:399)
at
shark.execution.FileSinkOperator.processPartition(FileSinkOperator.scala:72)
at
shark.execution.FileSinkOperator$.writeFiles$1(FileSinkOperator.scala:133)
at
shark.execution.FileSinkOperator$$anonfun$executeProcessFileSinkPartition$1.apply(FileSinkOperator.scala:138)
at
shark.execution.FileSinkOperator$$anonfun$executeProcessFileSinkPartition$1.apply(FileSinkOperator.scala:138)
at spark.scheduler.ResultTask.run(ResultTask.scala:77)
at spark.executor.Executor$TaskRunner.run(Executor.scala:98)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Here are some queries concerned:
CREATE TABLE hbase_byg_client (
idclient string,
isfictif boolean,
visites array < struct <
idvisite:string,datevisite:string,isauthent:boolean, affichages: array <
struct < page:string,idcategorie:int,freq:int >>>>)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" =
":key,navi:isfictif,navi:visites#s")
TBLPROPERTIES("hbase.table.name" = "byg_client")
;
INSERT OVERWRITE table hbase_byg_client
SELECT idClient, isfictif, collect_row(named_struct('idvisite',
idVisite, 'dateVisite', dateVisit, 'isAuthent', isAuthent, 'affichages',
t.affichages)) AS visites
FROM(
SELECT idClient, isfictif, idVisite, dateVisit, isAuthent,
collect_row(named_struct('page', page, 'IdCategorie', IdCategorie,
'freq', freq)) AS affichages
FROM v_byg_clean
GROUP BY idClient, isfictif, idVisite, dateVisit, isAuthent) t
GROUP BY idClient, isfictif
;
Actually, hbase_byg_client contains a complex non-primitive type field.
Any workaround here ?
Thank you.
Hao
Le 01/08/2013 21:00, lars hofhansl a écrit :
> Need to set scanner caching, otherwise each call to next will be an network RTT.
>
>
>
> ________________________________
> From: Hao Ren <h....@claravista.fr>
> To: user@hbase.apache.org
> Sent: Thursday, August 1, 2013 7:45 AM
> Subject: Why HBase integation with Hive makes Hive slow
>
>
> Hi,
>
> I have a cluster (1 master + 3 slaves) on which there Hive, Hbase, and
> Hadoop.
>
> In order to do some daily row-level update routine, we need to integrate
> Hbase with hive, but the performance is not good.
>
> E.g. There are 2 tables in hive,
> hbase_table: a hbase table created via Hive
> hive_table: a native hive table
> both hold the same data set.
>
> When runing:
> select count(*) from hbase_table; ===> takes 500 s
> select count(*) from hive_table; ===> takes 6 s
>
> I have tried a lot of queries on the two tables. But hbase_table is
> always very slow.
>
> To be claire, I created the hbase_ table as below:
>
> CREATE TABLE hbase_table (
> idvisite string,
> client_list Array<string>,
> nb_client int)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" =
> ":key,clients:id_list,clients:nb")
> TBLPROPERTIES("hbase.table.name" = "table_test")
> ;
>
> And my Hbase is on pseudo-distributed mode.
>
> I guess, at the beginning of a hive query execution, hive will load data
> from Hbase, where serde takes a long time.
>
> Could someone tell me how to improve my poor performance ?
> Is this cause by my wrongly configured integration ?
> Is a fully-distributed mode needed here ?
>
> Thank you in advance for your time.
>
> Hao.
>
>
--
Hao Ren
ClaraVista
www.claravista.fr
Re: Why HBase integation with Hive makes Hive slow
Posted by lars hofhansl <la...@apache.org>.
Need to set scanner caching, otherwise each call to next will be an network RTT.
________________________________
From: Hao Ren <h....@claravista.fr>
To: user@hbase.apache.org
Sent: Thursday, August 1, 2013 7:45 AM
Subject: Why HBase integation with Hive makes Hive slow
Hi,
I have a cluster (1 master + 3 slaves) on which there Hive, Hbase, and
Hadoop.
In order to do some daily row-level update routine, we need to integrate
Hbase with hive, but the performance is not good.
E.g. There are 2 tables in hive,
hbase_table: a hbase table created via Hive
hive_table: a native hive table
both hold the same data set.
When runing:
select count(*) from hbase_table; ===> takes 500 s
select count(*) from hive_table; ===> takes 6 s
I have tried a lot of queries on the two tables. But hbase_table is
always very slow.
To be claire, I created the hbase_ table as below:
CREATE TABLE hbase_table (
idvisite string,
client_list Array<string>,
nb_client int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" =
":key,clients:id_list,clients:nb")
TBLPROPERTIES("hbase.table.name" = "table_test")
;
And my Hbase is on pseudo-distributed mode.
I guess, at the beginning of a hive query execution, hive will load data
from Hbase, where serde takes a long time.
Could someone tell me how to improve my poor performance ?
Is this cause by my wrongly configured integration ?
Is a fully-distributed mode needed here ?
Thank you in advance for your time.
Hao.
--
Hao Ren
ClaraVista
www.claravista.fr