You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hao Ren <h....@claravista.fr> on 2013/08/01 16:45:16 UTC

Why HBase integation with Hive makes Hive slow

Hi,

I have a cluster (1 master + 3 slaves) on which there Hive, Hbase, and 
Hadoop.

In order to do some daily row-level update routine, we need to integrate 
Hbase with hive, but the performance is not good.

E.g. There are 2 tables in hive,
     hbase_table:  a hbase table created via Hive
     hive_table: a native hive table
  both hold the same data set.

When runing:
     select count(*) from hbase_table; ===> takes 500 s
     select count(*) from hive_table; ===> takes 6 s

I have tried a lot of queries on the two tables. But hbase_table is 
always very slow.

To be claire, I created the hbase_ table as below:

CREATE TABLE hbase_table (
idvisite string,
client_list Array<string>,
nb_client int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = 
":key,clients:id_list,clients:nb")
TBLPROPERTIES("hbase.table.name" = "table_test")
;

And my Hbase is on pseudo-distributed mode.

I guess, at the beginning of a hive query execution, hive will load data 
from Hbase, where serde takes a long time.

Could someone tell me how to improve my poor performance ?
Is this cause by my wrongly configured integration ?
Is a fully-distributed mode needed here ?

Thank you in advance for your time.

Hao.


-- 
Hao Ren
ClaraVista
www.claravista.fr

Re: Why HBase integation with Hive makes Hive slow

Posted by Andrew Purtell <ap...@apache.org>.
This is a NPE from Hive code.

But it looks like you are running Hive on Shark?

So this looks like an inquiry for the Shark mailing list.

Good luck!

On Fri, Aug 2, 2013 at 8:04 AM, Hao Ren <h....@claravista.fr> wrote:

> java.lang.NullPointerException
>     at org.apache.hadoop.hive.serde2.**objectinspector.primitive.**
> WritableIntObjectInspector.**get(**WritableIntObjectInspector.**java:35)
>     at org.apache.hadoop.hive.serde2.**lazy.LazyUtils.**
> writePrimitiveUTF8(LazyUtils.**java:199)
>     at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:696)
>     at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:758)
>     at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:713)
>     at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:758)
>     at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:713)
>     at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:685)
>     at org.apache.hadoop.hive.hbase.**HBaseSerDe.serializeField(**
> HBaseSerDe.java:648)
>     at org.apache.hadoop.hive.hbase.**HBaseSerDe.serialize(**
> HBaseSerDe.java:560)
>     at org.apache.hadoop.hive.ql.**exec.FileSinkOperator.**
> processOp(FileSinkOperator.**java:568)
>     at shark.execution.**FileSinkOperator$anonfun$**
> processPartition$1.apply(**FileSinkOperator.scala:73)
>     at shark.execution.**FileSinkOperator$anonfun$**
> processPartition$1.apply(**FileSinkOperator.scala:72)
>




-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Why HBase integation with Hive makes Hive slow

Posted by Hao Ren <h....@claravista.fr>.
Thank you, lars

The performance is largely improved when setting scanner caching to 10000
But I still encounter a problem.

When loading data to a hbast table via hive, I got a NullPointrExecption:

java.lang.NullPointerException
     at 
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:35)
     at 
org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:199)
     at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:696)
     at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:758)
     at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:713)
     at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:758)
     at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:713)
     at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:685)
     at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serializeField(HBaseSerDe.java:648)
     at 
org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:560)
     at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:568)
     at 
shark.execution.FileSinkOperator$$anonfun$processPartition$1.apply(FileSinkOperator.scala:73)
     at 
shark.execution.FileSinkOperator$$anonfun$processPartition$1.apply(FileSinkOperator.scala:72)
     at scala.collection.Iterator$class.foreach(Iterator.scala:772)
     at scala.collection.Iterator$$anon$19.foreach(Iterator.scala:399)
     at 
shark.execution.FileSinkOperator.processPartition(FileSinkOperator.scala:72)
     at 
shark.execution.FileSinkOperator$.writeFiles$1(FileSinkOperator.scala:133)
     at 
shark.execution.FileSinkOperator$$anonfun$executeProcessFileSinkPartition$1.apply(FileSinkOperator.scala:138)
     at 
shark.execution.FileSinkOperator$$anonfun$executeProcessFileSinkPartition$1.apply(FileSinkOperator.scala:138)
     at spark.scheduler.ResultTask.run(ResultTask.scala:77)
     at spark.executor.Executor$TaskRunner.run(Executor.scala:98)
     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:724)

Here are some queries concerned:

CREATE TABLE hbase_byg_client (
idclient string,
isfictif boolean,
visites array < struct < 
idvisite:string,datevisite:string,isauthent:boolean, affichages: array < 
struct < page:string,idcategorie:int,freq:int >>>>)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = 
":key,navi:isfictif,navi:visites#s")
TBLPROPERTIES("hbase.table.name" = "byg_client")
;

INSERT OVERWRITE table hbase_byg_client
SELECT idClient, isfictif, collect_row(named_struct('idvisite', 
idVisite, 'dateVisite', dateVisit, 'isAuthent', isAuthent, 'affichages', 
t.affichages)) AS visites
FROM(
   SELECT idClient, isfictif, idVisite, dateVisit, isAuthent, 
collect_row(named_struct('page', page, 'IdCategorie', IdCategorie, 
'freq', freq)) AS affichages
   FROM v_byg_clean
   GROUP BY idClient, isfictif, idVisite, dateVisit, isAuthent) t
GROUP BY idClient, isfictif
;

Actually, hbase_byg_client contains a complex non-primitive type field.

Any workaround here ?

Thank you.

Hao

Le 01/08/2013 21:00, lars hofhansl a écrit :
> Need to set scanner caching, otherwise each call to next will be an network RTT.
>
>
>
> ________________________________
>   From: Hao Ren <h....@claravista.fr>
> To: user@hbase.apache.org
> Sent: Thursday, August 1, 2013 7:45 AM
> Subject: Why HBase integation with Hive makes Hive slow
>   
>
> Hi,
>
> I have a cluster (1 master + 3 slaves) on which there Hive, Hbase, and
> Hadoop.
>
> In order to do some daily row-level update routine, we need to integrate
> Hbase with hive, but the performance is not good.
>
> E.g. There are 2 tables in hive,
>       hbase_table:  a hbase table created via Hive
>       hive_table: a native hive table
>    both hold the same data set.
>
> When runing:
>       select count(*) from hbase_table; ===> takes 500 s
>       select count(*) from hive_table; ===> takes 6 s
>
> I have tried a lot of queries on the two tables. But hbase_table is
> always very slow.
>
> To be claire, I created the hbase_ table as below:
>
> CREATE TABLE hbase_table (
> idvisite string,
> client_list Array<string>,
> nb_client int)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES ("hbase.columns.mapping" =
> ":key,clients:id_list,clients:nb")
> TBLPROPERTIES("hbase.table.name" = "table_test")
> ;
>
> And my Hbase is on pseudo-distributed mode.
>
> I guess, at the beginning of a hive query execution, hive will load data
> from Hbase, where serde takes a long time.
>
> Could someone tell me how to improve my poor performance ?
> Is this cause by my wrongly configured integration ?
> Is a fully-distributed mode needed here ?
>
> Thank you in advance for your time.
>
> Hao.
>
>


-- 
Hao Ren
ClaraVista
www.claravista.fr

Re: Why HBase integation with Hive makes Hive slow

Posted by lars hofhansl <la...@apache.org>.
Need to set scanner caching, otherwise each call to next will be an network RTT.



________________________________
 From: Hao Ren <h....@claravista.fr>
To: user@hbase.apache.org 
Sent: Thursday, August 1, 2013 7:45 AM
Subject: Why HBase integation with Hive makes Hive slow
 

Hi,

I have a cluster (1 master + 3 slaves) on which there Hive, Hbase, and 
Hadoop.

In order to do some daily row-level update routine, we need to integrate 
Hbase with hive, but the performance is not good.

E.g. There are 2 tables in hive,
     hbase_table:  a hbase table created via Hive
     hive_table: a native hive table
  both hold the same data set.

When runing:
     select count(*) from hbase_table; ===> takes 500 s
     select count(*) from hive_table; ===> takes 6 s

I have tried a lot of queries on the two tables. But hbase_table is 
always very slow.

To be claire, I created the hbase_ table as below:

CREATE TABLE hbase_table (
idvisite string,
client_list Array<string>,
nb_client int)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = 
":key,clients:id_list,clients:nb")
TBLPROPERTIES("hbase.table.name" = "table_test")
;

And my Hbase is on pseudo-distributed mode.

I guess, at the beginning of a hive query execution, hive will load data 
from Hbase, where serde takes a long time.

Could someone tell me how to improve my poor performance ?
Is this cause by my wrongly configured integration ?
Is a fully-distributed mode needed here ?

Thank you in advance for your time.

Hao.


-- 
Hao Ren
ClaraVista
www.claravista.fr