You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Christian Schäfer <sy...@yahoo.de> on 2011/10/14 15:26:37 UTC
Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Hi there,
just saw that there is a JDBC-Driver for HBase.
So I wonder if it's possible to persist to HBase via JPA using that JDBC-Driver?
For now I still saw in the examples that there are Annotations included in HBql what seems not so far away from JPA-Annotations.
JPA-HBql-. Examples (I didnt find any) would be welcome :-)
regards
Christian
Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Posted by Christian Schäfer <sy...@yahoo.de>.
Hi Frédéric,
Thanks for you info ...and sorry for the late answer...
I'm currently waiting for a test system...could still take a few weeks.. :-(
When its up and running I will start to compare at first compare hbase raw api vs. data nucleus-orm on different db-sizes (20GB - 100GB)
If datanucleus-orm is much slower than hbase-api I will test the other ones you mentioned ;)
Should someone be interested in the results I willl post them here ..
Somehow I wonder that there hasn't been such benchmarks yet....or have I just missed them?
regards
Chris
________________________________
Von: Frédéric Fondement <fr...@uha.fr>
An: user@hbase.apache.org
Gesendet: 10:24 Dienstag, 18.Oktober 2011
Betreff: Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Le 15/10/11 23:34, Christian Schäfer a écrit :
> But nevertheless I will try on using data nucleus' jpa for hbase and make some benchmarks to compare it with the hbase native interface;-)
Hi there,
Would be great if you plan to make such study to publish results (here ?) !!!
What about proposing a simple application that all those guys who created an ORM (Datanucleus, Kundera, ...) could implement and submit (you?) for a bench ?
I'm part of those guys. We created n-orm (http://code.google.com/p/n-orm/) just as a matter to separate responsibilities in our team (functionnal vs non-functionnal), to centralize data management (to improve separation of concerns, and thus maintainability), and to still understand what really happens under the hood (and still be able to change platform in case of problem...). Actually, our ORM considers POJOs as some kind of schema for the base (query-driven), and thus, philosophy is more to use java objects but with the knowledge of how to use HBase in mind, so that we hope not loosing too much of HBase possibilities.
I agree when Michel says that the HBase API is easy, but when it comes to details, it's really hard to think of everything, especially when it's interleaved with functionnal code (scan caching, inter-process schema management, compression, migration, error handling, new versions of the API, new possibilities... or just learning a new important stuff to be integrated in the complete application !).
Nevertheless, as our application becomes more and more complex, it's unconceivable for us to re-implement it just using the HBase raw API. But, as a consequence, I have no real idea of the price we pay regarding performance just to help us developing...
Another ORM that deserves attention is https://github.com/ghelmling/meetup.beeno which is built on the same philosophy. Actually, we didn't choose it as it's too tightly coupled with HBase, but I guess it must really perform well (because of the latter reason).
I think the real danger of ORMs is to think your schema in a domain-driven (classical) fashion, instead of query-driven. It might be the case that this danger is less important when you use raw APIs.
Cheers,
Frédéric.
Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Posted by Frédéric Fondement <fr...@uha.fr>.
Le 15/10/11 23:34, Christian Schäfer a écrit :
> But nevertheless I will try on using data nucleus' jpa for hbase and make some benchmarks to compare it with the hbase native interface;-)
Hi there,
Would be great if you plan to make such study to publish results (here
?) !!!
What about proposing a simple application that all those guys who
created an ORM (Datanucleus, Kundera, ...) could implement and submit
(you?) for a bench ?
I'm part of those guys. We created n-orm
(http://code.google.com/p/n-orm/) just as a matter to separate
responsibilities in our team (functionnal vs non-functionnal), to
centralize data management (to improve separation of concerns, and thus
maintainability), and to still understand what really happens under the
hood (and still be able to change platform in case of problem...).
Actually, our ORM considers POJOs as some kind of schema for the base
(query-driven), and thus, philosophy is more to use java objects but
with the knowledge of how to use HBase in mind, so that we hope not
loosing too much of HBase possibilities.
I agree when Michel says that the HBase API is easy, but when it comes
to details, it's really hard to think of everything, especially when
it's interleaved with functionnal code (scan caching, inter-process
schema management, compression, migration, error handling, new versions
of the API, new possibilities... or just learning a new important stuff
to be integrated in the complete application !).
Nevertheless, as our application becomes more and more complex, it's
unconceivable for us to re-implement it just using the HBase raw API.
But, as a consequence, I have no real idea of the price we pay regarding
performance just to help us developing...
Another ORM that deserves attention is
https://github.com/ghelmling/meetup.beeno which is built on the same
philosophy. Actually, we didn't choose it as it's too tightly coupled
with HBase, but I guess it must really perform well (because of the
latter reason).
I think the real danger of ORMs is to think your schema in a
domain-driven (classical) fashion, instead of query-driven. It might be
the case that this danger is less important when you use raw APIs.
Cheers,
Frédéric.
Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Posted by Christian Schäfer <sy...@yahoo.de>.
Ok..so if using jpa would kill hbase's performance...it may be an idea to use hbase native interface for hbase and jpa only for traditional DBs...so there are two interfaces..
But nevertheless I will try on using data nucleus' jpa for hbase and make some benchmarks to compare it with the hbase native interface ;-)
Btw. I don't think I would have to forget M/R as I would only use JPA to get data into hbase or to get few records out of it....for analysis via M/R I would of course not use any JPA stuff but the native way..
comments welcome..as I'm a greenhorn to hadoop concepts :)
________________________________
Von: Michael Segel <mi...@hotmail.com>
An: syrious3000@yahoo.de
Gesendet: 22:27 Samstag, 15.Oktober 2011
Betreff: RE: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Ok. So scratch the M/R then.
I would still suggest that you really don't try and create a single interface. You're going to reduce your performance unnecessarily and not gain much in the process. HBase doesn't fit well in the JDBC realm. There is no such concept of a transaction.
If you're going to try and solve a problem w two totally different types of data stores, you may want to get a little more creative. But trying to force HBase's api into a JPA which was designed for RDBMs... not really a good idea.
JMHO, YMMV.
-Mike
________________________________
Date: Sat, 15 Oct 2011 20:41:09 +0100
From: syrious3000@yahoo.de
Subject: Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
To: michael_segel@hotmail.com
CC: user@hbase.apache.org
Yes, the hbase access API may be easy but in the current project I want to have an exchangeable persistence solution without needing to write adapter-code for each db.
That's because some customers of the system will have huge loads of data, so they will use hbase...for other small customers traditional DBs would be enough..
So the only requirement is a jpable database ;)
If I missunderstood sth..please tell me :-)
________________________________
Von: Michel Segel <mi...@hotmail.com>
An: "user@hbase.apache.org" <us...@hbase.apache.org>
Gesendet: 17:18 Samstag, 15.Oktober 2011
Betreff: Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
<mini rant>
ok...
There are two ways to use HBase. One is as part of a map reduce program. The other is as a NoSQL data store.
If you are writing a m/r job, you really don't want to create any DAO layer or even think about a JPA layer. Sorry but the access API is too simple to require any JPA and you're not writing to multiple data sources so you really don't want to do any abstraction...
Keep It Simple.
If you are writing an app to access HBase outside of a map reduce job,
most of the same arguments apply. HBase access patterns are too simple to require any abstraction.
</mini rant>
Sorry,
JMHO
YMMV, so take what I say with a grain of salt.
Sent from a remote device. Please excuse any typos...
Mike Segel
On Oct 14, 2011, at 3:54 PM, Christian Schäfer <sy...@yahoo.de> wrote:
> Hi St.Ack
>
> yes I'm currently using Data Nucleus...and thought I missed that there is a JDBC-Driver
>
>
>
>
> ________________________________
> Von: Stack <st...@duboce.net>
> An: user@hbase.apache.org; Christian Schäfer <sy...@yahoo.de>
> Gesendet: 18:35 Freitag, 14.Oktober 2011
> Betreff: Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
>
> On Fri, Oct 14, 2011 at 1:26 PM, Christian Schäfer <sy...@yahoo.de> wrote:
>> just saw that there is a JDBC-Driver for HBase.
>>
>> So I wonder if it's possible to persist to HBase via JPA using that JDBC-Driver?
>>
>> For now I still saw in the examples that there are Annotations included in HBql what seems not so far away from JPA-Annotations.
>>
>> JPA-HBql-. Examples (I didnt find any) would be welcome :-)
>>
>
> I think you might need to temper your expectation a little. I think
> that jdbc driver is for hbql -- least thats what it
understands.
> Then I'm not sure of the state of hbql.
>
> The above seems a little dodgy to be building a jpa bridge upon? If
> you google jpa and hbase there seem to be a few possibilities
> (datanucleus for one).
>
> St.Ack
Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Posted by Christian Schäfer <sy...@yahoo.de>.
Yes, the hbase access API may be easy but in the current project I want to have an exchangeable persistence solution without needing to write adapter-code for each db.
That's because some customers of the system will have huge loads of data, so they will use hbase...for other small customers traditional DBs would be enough..
So the only requirement is a jpable database ;)
If I missunderstood sth..please tell me :-)
________________________________
Von: Michel Segel <mi...@hotmail.com>
An: "user@hbase.apache.org" <us...@hbase.apache.org>
Gesendet: 17:18 Samstag, 15.Oktober 2011
Betreff: Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
<mini rant>
ok...
There are two ways to use HBase. One is as part of a map reduce program. The other is as a NoSQL data store.
If you are writing a m/r job, you really don't want to create any DAO layer or even think about a JPA layer. Sorry but the access API is too simple to require any JPA and you're not writing to multiple data sources so you really don't want to do any abstraction...
Keep It Simple.
If you are writing an app to access HBase outside of a map reduce job, most of the same arguments apply. HBase access patterns are too simple to require any abstraction.
</mini rant>
Sorry,
JMHO
YMMV, so take what I say with a grain of salt.
Sent from a remote device. Please excuse any typos...
Mike Segel
On Oct 14, 2011, at 3:54 PM, Christian Schäfer <sy...@yahoo.de> wrote:
> Hi St.Ack
>
> yes I'm currently using Data Nucleus...and thought I missed that there is a JDBC-Driver
>
>
>
>
> ________________________________
> Von: Stack <st...@duboce.net>
> An: user@hbase.apache.org; Christian Schäfer <sy...@yahoo.de>
> Gesendet: 18:35 Freitag, 14.Oktober 2011
> Betreff: Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
>
> On Fri, Oct 14, 2011 at 1:26 PM, Christian Schäfer <sy...@yahoo.de> wrote:
>> just saw that there is a JDBC-Driver for HBase.
>>
>> So I wonder if it's possible to persist to HBase via JPA using that JDBC-Driver?
>>
>> For now I still saw in the examples that there are Annotations included in HBql what seems not so far away from JPA-Annotations.
>>
>> JPA-HBql-. Examples (I didnt find any) would be welcome :-)
>>
>
> I think you might need to temper your expectation a little. I think
> that jdbc driver is for hbql -- least thats what it understands.
> Then I'm not sure of the state of hbql.
>
> The above seems a little dodgy to be building a jpa bridge upon? If
> you google jpa and hbase there seem to be a few possibilities
> (datanucleus for one).
>
> St.Ack
Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Posted by Michel Segel <mi...@hotmail.com>.
<mini rant>
ok...
There are two ways to use HBase. One is as part of a map reduce program. The other is as a NoSQL data store.
If you are writing a m/r job, you really don't want to create any DAO layer or even think about a JPA layer. Sorry but the access API is too simple to require any JPA and you're not writing to multiple data sources so you really don't want to do any abstraction...
Keep It Simple.
If you are writing an app to access HBase outside of a map reduce job, most of the same arguments apply. HBase access patterns are too simple to require any abstraction.
</mini rant>
Sorry,
JMHO
YMMV, so take what I say with a grain of salt.
Sent from a remote device. Please excuse any typos...
Mike Segel
On Oct 14, 2011, at 3:54 PM, Christian Schäfer <sy...@yahoo.de> wrote:
> Hi St.Ack
>
> yes I'm currently using Data Nucleus...and thought I missed that there is a JDBC-Driver
>
>
>
>
> ________________________________
> Von: Stack <st...@duboce.net>
> An: user@hbase.apache.org; Christian Schäfer <sy...@yahoo.de>
> Gesendet: 18:35 Freitag, 14.Oktober 2011
> Betreff: Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
>
> On Fri, Oct 14, 2011 at 1:26 PM, Christian Schäfer <sy...@yahoo.de> wrote:
>> just saw that there is a JDBC-Driver for HBase.
>>
>> So I wonder if it's possible to persist to HBase via JPA using that JDBC-Driver?
>>
>> For now I still saw in the examples that there are Annotations included in HBql what seems not so far away from JPA-Annotations.
>>
>> JPA-HBql-. Examples (I didnt find any) would be welcome :-)
>>
>
> I think you might need to temper your expectation a little. I think
> that jdbc driver is for hbql -- least thats what it understands.
> Then I'm not sure of the state of hbql.
>
> The above seems a little dodgy to be building a jpa bridge upon? If
> you google jpa and hbase there seem to be a few possibilities
> (datanucleus for one).
>
> St.Ack
Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Posted by Christian Schäfer <sy...@yahoo.de>.
Hi St.Ack
yes I'm currently using Data Nucleus...and thought I missed that there is a JDBC-Driver
________________________________
Von: Stack <st...@duboce.net>
An: user@hbase.apache.org; Christian Schäfer <sy...@yahoo.de>
Gesendet: 18:35 Freitag, 14.Oktober 2011
Betreff: Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
On Fri, Oct 14, 2011 at 1:26 PM, Christian Schäfer <sy...@yahoo.de> wrote:
> just saw that there is a JDBC-Driver for HBase.
>
> So I wonder if it's possible to persist to HBase via JPA using that JDBC-Driver?
>
> For now I still saw in the examples that there are Annotations included in HBql what seems not so far away from JPA-Annotations.
>
> JPA-HBql-. Examples (I didnt find any) would be welcome :-)
>
I think you might need to temper your expectation a little. I think
that jdbc driver is for hbql -- least thats what it understands.
Then I'm not sure of the state of hbql.
The above seems a little dodgy to be building a jpa bridge upon? If
you google jpa and hbase there seem to be a few possibilities
(datanucleus for one).
St.Ack
Re: Persist to HBase with JPA using HBql-JDBC-Driver (Examples)?
Posted by Stack <st...@duboce.net>.
On Fri, Oct 14, 2011 at 1:26 PM, Christian Schäfer <sy...@yahoo.de> wrote:
> just saw that there is a JDBC-Driver for HBase.
>
> So I wonder if it's possible to persist to HBase via JPA using that JDBC-Driver?
>
> For now I still saw in the examples that there are Annotations included in HBql what seems not so far away from JPA-Annotations.
>
> JPA-HBql-. Examples (I didnt find any) would be welcome :-)
>
I think you might need to temper your expectation a little. I think
that jdbc driver is for hbql -- least thats what it understands.
Then I'm not sure of the state of hbql.
The above seems a little dodgy to be building a jpa bridge upon? If
you google jpa and hbase there seem to be a few possibilities
(datanucleus for one).
St.Ack