Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Nick maillard <ni...@fifty-five.com>]

Hi everyone

Continuing on my journey in Hadoop world.
I have installed:
Hive 0.9 on
Hbase 0.94.2 cluster atop
an Hadoop 1.0.3 cluster.

When I do simply query like "select *" everything is fine.
When I try "select foo from myTestTable" I see the MR 
start on my hadoop monitor UI and it fails with:

Error: java.lang.ClassNotFoundException: 
com.google.protobuf.Message
at java.net.URLClassLoader$1.run
at java.net.URLClassLoader$1.run
at java.security.AccessController.doPrivileged
at java.net.URLClassLoader.findClass
at java.lang.ClassLoader.loadClass
at sun.misc.Launcher$AppClassLoader.loadClass
at java.lang.ClassLoader.loadClass
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.<clinit
at org.apache.hadoop.hbase.ipc.Invocation.write

Anyone have this problem: 
The jar is present in hbase lib directory.
I have copied the jar in my hive libs directory with no result.
 

Re: Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Nick maillard <ni...@fifty-five.com>]

Hi Jean-Daniel

We are trying different types of software to solve our size issue.
We aggregate data on website interactions and this can easily go up to a couple
of millions lines with a couple tens of interactions types per day. We would
like to keep a rolling 2 years history. We expect this will grow in time as this
is the current situation.

We have queries that must anwser very rapidly on this set for example uses that
did interaction A and/or B in the last week and that have a profile C and are
currently on page Y. This typically should be realtime, either different Hbase
tables or fine tuned indexes.

For statistical purposes we have longer queries where we will, a of couple times
a day or week extract relevant susbsets and apply differents types of queries
depending on the need. Some are more regular and some are on customer inquiries.
These susbsets can be extracted and calculated to populate an index or serve as
raw data for open research and analysis.

>From what I understand Hbase is not an "open query" system but must be designed
with the need and question in mind. For other purposes I'd like to use Habse as
a long term data storage and extract relevant susbsets when needed to populate
other systems that will do the job needed.

Does this seem a rationnal approach? 

Re: Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Jean-Daniel Cryans <jd...@apache.org>]

On Thu, Oct 25, 2012 at 9:00 AM, Nick maillard
<ni...@fifty-five.com> wrote:
> Hi Jean-Daniel
>
> Again thanks for the quick reply and for the env detail I'll get to it.
>
> Of course select count (*) is not what I want to optimize.
> My more regular queries will have an Hbase schema designed for them using the
> rowkeys and potentially column families etc...
> I'm guessing Hive uses the rowkey hash aspect when in the sql query.

HBase row keys aren't hashes, it relies completely on their
lexicographical sorted nature. Hive really just uses HBase's input
format, which creates 1 map per region. Then each mapper scans from
the start key to the end key within each region in parallel with the
other mappers.

>
> My question on a more general note. When querying hbase through hive on tables
> that have not been designed specifically with that typeof query in mind I wanted
> to keep query time low. I'm trying to get a feel of when I should make table
> with a thought out rowkey, family etc.. and to what extent I can have a decent
> query time on more exotic queries.

What kind of "decent query time" are you looking for?

>
> I am trying to decide If I make several tables on a dataset for the very common
> queries and for other more rare queries If Hive can give me good resolve time or
> If I should use to extract a good view to feed to other querying systems, like
> big query or Mysql or anything.

It really depends on what your use case is.

Re: Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Nick maillard <ni...@fifty-five.com>]

Hi Jean-Daniel

Again thanks for the quick reply and for the env detail I'll get to it.

Of course select count (*) is not what I want to optimize.
My more regular queries will have an Hbase schema designed for them using the
rowkeys and potentially column families etc...
I'm guessing Hive uses the rowkey hash aspect when in the sql query.

My question on a more general note. When querying hbase through hive on tables
that have not been designed specifically with that typeof query in mind I wanted
to keep query time low. I'm trying to get a feel of when I should make table
with a thought out rowkey, family etc.. and to what extent I can have a decent
query time on more exotic queries.

I am trying to decide If I make several tables on a dataset for the very common
queries and for other more rare queries If Hive can give me good resolve time or
If I should use to extract a good view to feed to other querying systems, like
big query or Mysql or anything.

If have read http://hbase.apache.org/book.html#performance but off course as
implement I get a better understanding and I need reread all documentation
regularly.

Thanks for your time and answers
 

Re: Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Jean-Daniel Cryans <jd...@apache.org>]

On Thu, Oct 25, 2012 at 8:31 AM, Nick maillard
<ni...@fifty-five.com> wrote:
> Hi jean-Daniel
>
> Ok I'll sent it in the env thanks for the advice.
> Are their other libs I might need to add?

The usual client libs... doesn't seem like we documented them
anywhere... it's pretty much what you have in now.

> Could just tell hive to use it's lib directory or hbase's lib directory in it's
> classpath in some way?

That's a question for the hive ML.

> I could just set it in the bashrc but that's not very elegant.

I really meant that you should use HIVE_AUX_JARS_PATH in hive-env.sh

>
> Another thing I am testing my 3 machine hadoop cluster.
> I have queried 'select * from myTestTable' which has 1719428 entries.
> The 7 map tasks and 1 reducer took almost 5 minutes to compute, I am right to
> think it is a little slow?

You have a 1-2 minutes overhead in there because you are using
MapReduce, then usually one should set hbase.client.scanner.caching to
a better value than 1. It's client-side so hive needs to have it. But
everything will seem slow when using MR on such a small dataset, a
single client running a scan would be faster in this case.

> How could I make this go faster, more map tasks, more nodes?

Is select count(*) really the use case you want to optimize? Have you
read this? http://hbase.apache.org/book.html#performance

>
> True I would never scan a whole table usually but I could easily have queries
> that MR over a set of this size.
>

Re: Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Nick maillard <ni...@fifty-five.com>]

Sorry typo my query was 'select count(*) from myTestTable'

Re: Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Nick maillard <ni...@fifty-five.com>]

Hi jean-Daniel

Ok I'll sent it in the env thanks for the advice.
Are their other libs I might need to add?
Could just tell hive to use it's lib directory or hbase's lib directory in it's
classpath in some way?
I could just set it in the bashrc but that's not very elegant.

Another thing I am testing my 3 machine hadoop cluster.
I have queried 'select * from myTestTable' which has 1719428 entries.
The 7 map tasks and 1 reducer took almost 5 minutes to compute, I am right to
think it is a little slow?
How could I make this go faster, more map tasks, more nodes?

True I would never scan a whole table usually but I could easily have queries
that MR over a set of this size.

Re: Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Jean-Daniel Cryans <jd...@apache.org>]

Oh I meant to say protobuf, not guava. And yes that's how it should be
done. You can also set your auxpath in your hive-env so that you don't
need to keep a big command line around.

J-D

On Thu, Oct 25, 2012 at 8:06 AM, Nick maillard
<ni...@fifty-five.com> wrote:
> Hi Jean-Daniel
>
> Thanks for the quick reply.
> Guava is already in my auxpath.
>
> I guess I have found a solution by adding protobuff to the auxpath.
>
> Basically instead of:
> ./bin/hive --auxpath
> $HIVE_HOME/lib/hive-hbase-handler-0.9.0.jar,
> $HBASE_HOME/hbase-0.94.2.jar,
> $HBASE_HOME/lib/zookeeper-3.4.3.jar,
> $HBASE_HOME/lib/guava-11.0.2.jar
>
> I have added at the end
> $HBASE_HOME/lib/protobuf-java-2.4.0a.jar at the end
>
> it works but I'm sure it's not a good way of going about this.
> Maybe I have not put something in the hive-site.xml.
> If anyone knows what I might have done wrong.
>
>

Re: Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Nick maillard <ni...@fifty-five.com>]

Hi Jean-Daniel

Thanks for the quick reply. 
Guava is already in my auxpath. 

I guess I have found a solution by adding protobuff to the auxpath.

Basically instead of:
./bin/hive --auxpath
$HIVE_HOME/lib/hive-hbase-handler-0.9.0.jar,
$HBASE_HOME/hbase-0.94.2.jar,
$HBASE_HOME/lib/zookeeper-3.4.3.jar,
$HBASE_HOME/lib/guava-11.0.2.jar

I have added at the end
$HBASE_HOME/lib/protobuf-java-2.4.0a.jar at the end

it works but I'm sure it's not a good way of going about this.
Maybe I have not put something in the hive-site.xml.
If anyone knows what I might have done wrong.

Re: Hive Hbase 0.94 ClassNotFoundException com.google.protobuf.Message

[Jean-Daniel Cryans <jd...@apache.org>]

Hi Nick,

Have you tried what's in this documentation?

https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration#HBaseIntegration-Usage

It seems to me that you should add guava in the auxpath.

J-D

On Thu, Oct 25, 2012 at 7:52 AM, Nick maillard
<ni...@fifty-five.com> wrote:
> Hi everyone
>
> Continuing on my journey in Hadoop world.
> I have installed:
> Hive 0.9 on
> Hbase 0.94.2 cluster atop
> an Hadoop 1.0.3 cluster.
>
> When I do simply query like "select *" everything is fine.
> When I try "select foo from myTestTable" I see the MR
> start on my hadoop monitor UI and it fails with:
>
> Error: java.lang.ClassNotFoundException:
> com.google.protobuf.Message
> at java.net.URLClassLoader$1.run
> at java.net.URLClassLoader$1.run
> at java.security.AccessController.doPrivileged
> at java.net.URLClassLoader.findClass
> at java.lang.ClassLoader.loadClass
> at sun.misc.Launcher$AppClassLoader.loadClass
> at java.lang.ClassLoader.loadClass
> at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.<clinit
> at org.apache.hadoop.hbase.ipc.Invocation.write
>
> Anyone have this problem:
> The jar is present in hbase lib directory.
> I have copied the jar in my hive libs directory with no result.
>
>