You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jignesh Patel <ji...@websoft.com> on 2011/09/20 21:57:11 UTC
Using HBase for real time transaction
We are exploring possibility of using HBase for the real time transactions. Is that possible?
-Jignesh
RE: Using HBase for real time transaction
Posted by "GOEKE, MATTHEW (AG/1000)" <ma...@monsanto.com>.
In order to answer you first question we would need to know what types of data you plan on storing and latency requirements. If it is semistructured/unstructured data then HBase *can* be a great fit but I have seen very few cases where you will want to scrap your RDBMS completely. Most organizations that use HBase will still have a need for a RDBMS/MPP solution for real time access to structured data.
Matt
-----Original Message-----
From: Jignesh Patel [mailto:jignesh@websoft.com]
Sent: Tuesday, September 20, 2011 4:25 PM
To: common-user@hadoop.apache.org
Subject: Re: Using HBase for real time transaction
Tom,
Let me reword: can HBase be used as a transactional database(i.e. in replacement of mysql)?
The requirement is to have real time read and write operations. I mean as soon as data is written the user should see the data(Here data should be written in Hbase).
-Jignesh
On Sep 20, 2011, at 5:11 PM, Tom Deutsch wrote:
> Real-time means different things to different people. Can you share your
> latency requirements from the time the data is generated to when it needs
> to be consumed, or how you are thinking of using Hbase in the overall
> flow?
>
> ------------------------------------------------
> Tom Deutsch
> Program Director
> CTO Office: Information Management
> Hadoop Product Manager / Customer Exec
> IBM
> 3565 Harbor Blvd
> Costa Mesa, CA 92626-1420
> tdeutsch@us.ibm.com
>
>
>
>
> Jignesh Patel <ji...@websoft.com>
> 09/20/2011 12:57 PM
> Please respond to
> common-user@hadoop.apache.org
>
>
> To
> common-user@hadoop.apache.org
> cc
>
> Subject
> Using HBase for real time transaction
>
>
>
>
>
>
> We are exploring possibility of using HBase for the real time
> transactions. Is that possible?
>
> -Jignesh
>
This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.
All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.
The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.
Re: Using HBase for real time transaction
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Wed, Sep 21, 2011 at 8:36 AM, Jignesh Patel <ji...@websoft.com> wrote:
> I am not looking for relational database. But looking creating multi tenant database, now at this time I am not sure whether it needs transactions or not and even that kind of architecture can support transactions.
Currently in HBase nothing prevents you from having multiple tenants,
as long as they have different table names. Also keep in mind that
there's no security implemented, but it *might* make it for 0.92
(crossing fingers).
>> Row mutations in HBase are seen by the user as soon as they are done,
>> atomicity is guaranteed at the row level, which seems to satisfy his
>> requirement. If multi-row transactions are needed then I agree HBase
>> might not be what he wants.
>
> Can't we handle transaction through application or container, before data even goes to HBase?
Sure, you could do something like what Megastore[1] does, but you
really need to evaluate your needs and see if that works.
>
> And I do have one more doubt, how to handle low read latency?
>
HBase offers that out of the box, a more precise question would be
what 99th percentile read latency you need. Just for the sake of
giving a data point, right now our 99p is 20ms but that's with our
type of workload, machines, front end caching, etc, so YYMV.
J-D
1. Megastore (transactions are described in chapter 3.3):
http://www.cidrdb.org/cidr2011/Papers/CIDR11_Paper32.pdf
Re: Using HBase for real time transaction
Posted by Jignesh Patel <ji...@websoft.com>.
On Sep 20, 2011, at 10:06 PM, Jean-Daniel Cryans wrote:
>> I think there has to be some clarification.
>>
>> The OP was asking about a mySQL replacement.
>> HBase will never be a RDBMS replacement. No Transactions means no way of doing OLTP.
>> Its the wrong tool for that type of work.
>
> Agreed, if you are looking to handle relational data in a relational
> fashion, might be better to look elsewhere
I am not looking for relational database. But looking creating multi tenant database, now at this time I am not sure whether it needs transactions or not and even that kind of architecture can support transactions.
>
>> Recognize what HBase is and what it is not.
>
> Not sure what you're referring to here.
>
>> This doesn't mean you can't take in or deliver data in real time, it can.
>> So if you want to use it in a real time manner, sure. Note that like with other databases, you will have to do some work to handle real time data.
>> I guess you would have to provide a specific use case on what you want to achieve in order to know if its a good fit.
>
> He says:
Hope above line explains what I am interested(multi tenant database)
>
>>> The requirement is to have real time read and write operations. I mean as soon as data is written the user should see the data(Here data should be written in Hbase).
>
> Row mutations in HBase are seen by the user as soon as they are done,
> atomicity is guaranteed at the row level, which seems to satisfy his
> requirement. If multi-row transactions are needed then I agree HBase
> might not be what he wants.
Can't we handle transaction through application or container, before data even goes to HBase?
And I do have one more doubt, how to handle low read latency?
-Jignesh
Re: Using HBase for real time transaction
Posted by Jean-Daniel Cryans <jd...@apache.org>.
> I think there has to be some clarification.
>
> The OP was asking about a mySQL replacement.
> HBase will never be a RDBMS replacement. No Transactions means no way of doing OLTP.
> Its the wrong tool for that type of work.
Agreed, if you are looking to handle relational data in a relational
fashion, might be better to look elsewhere
> Recognize what HBase is and what it is not.
Not sure what you're referring to here.
> This doesn't mean you can't take in or deliver data in real time, it can.
> So if you want to use it in a real time manner, sure. Note that like with other databases, you will have to do some work to handle real time data.
> I guess you would have to provide a specific use case on what you want to achieve in order to know if its a good fit.
He says:
>> The requirement is to have real time read and write operations. I mean as soon as data is written the user should see the data(Here data should be written in Hbase).
Row mutations in HBase are seen by the user as soon as they are done,
atomicity is guaranteed at the row level, which seems to satisfy his
requirement. If multi-row transactions are needed then I agree HBase
might not be what he wants.
J-D
RE: Using HBase for real time transaction
Posted by Michael Segel <mi...@hotmail.com>.
> Date: Tue, 20 Sep 2011 15:05:31 -0700
> Subject: Re: Using HBase for real time transaction
> From: jdcryans@apache.org
> To: common-user@hadoop.apache.org
>
> While HBase isn't ACID-compliant, it does have have some guarantees:
>
> http://hbase.apache.org/acid-semantics.html
>
> J-D
>
I think there has to be some clarification.
The OP was asking about a mySQL replacement.
HBase will never be a RDBMS replacement. No Transactions means no way of doing OLTP.
Its the wrong tool for that type of work.
Sure I know I can kludge something but its not worth the effort. Choose a better tool like a real database... e.g. Informix.
Recognize what HBase is and what it is not.
This doesn't mean you can't take in or deliver data in real time, it can.
So if you want to use it in a real time manner, sure. Note that like with other databases, you will have to do some work to handle real time data.
I guess you would have to provide a specific use case on what you want to achieve in order to know if its a good fit.
HTH
-Mike
> On Tue, Sep 20, 2011 at 2:56 PM, Michael Segel
> <mi...@hotmail.com> wrote:
> >
> > Since Tom isn't technical... ;-)
> >
> > The short answer is No.
> > HBase is not capable of being a transactional because it doesn't support transactions.
> > Nor is HBase ACID compliant.
> >
> > Having said that, yes you can use HBase to serve data in real time.
> >
> > HTH
> >
> > -Mike
> >
> >
Re: Using HBase for real time transaction
Posted by Jean-Daniel Cryans <jd...@apache.org>.
While HBase isn't ACID-compliant, it does have have some guarantees:
http://hbase.apache.org/acid-semantics.html
J-D
On Tue, Sep 20, 2011 at 2:56 PM, Michael Segel
<mi...@hotmail.com> wrote:
>
> Since Tom isn't technical... ;-)
>
> The short answer is No.
> HBase is not capable of being a transactional because it doesn't support transactions.
> Nor is HBase ACID compliant.
>
> Having said that, yes you can use HBase to serve data in real time.
>
> HTH
>
> -Mike
>
>
>> Subject: Re: Using HBase for real time transaction
>> From: jignesh@websoft.com
>> Date: Tue, 20 Sep 2011 17:25:17 -0400
>> To: common-user@hadoop.apache.org
>>
>> Tom,
>> Let me reword: can HBase be used as a transactional database(i.e. in replacement of mysql)?
>>
>> The requirement is to have real time read and write operations. I mean as soon as data is written the user should see the data(Here data should be written in Hbase).
>>
>> -Jignesh
>>
>>
>> On Sep 20, 2011, at 5:11 PM, Tom Deutsch wrote:
>>
>> > Real-time means different things to different people. Can you share your
>> > latency requirements from the time the data is generated to when it needs
>> > to be consumed, or how you are thinking of using Hbase in the overall
>> > flow?
>> >
>> > ------------------------------------------------
>> > Tom Deutsch
>> > Program Director
>> > CTO Office: Information Management
>> > Hadoop Product Manager / Customer Exec
>> > IBM
>> > 3565 Harbor Blvd
>> > Costa Mesa, CA 92626-1420
>> > tdeutsch@us.ibm.com
>> >
>> >
>> >
>> >
>> > Jignesh Patel <ji...@websoft.com>
>> > 09/20/2011 12:57 PM
>> > Please respond to
>> > common-user@hadoop.apache.org
>> >
>> >
>> > To
>> > common-user@hadoop.apache.org
>> > cc
>> >
>> > Subject
>> > Using HBase for real time transaction
>> >
>> >
>> >
>> >
>> >
>> >
>> > We are exploring possibility of using HBase for the real time
>> > transactions. Is that possible?
>> >
>> > -Jignesh
>> >
>>
>
RE: Using HBase for real time transaction
Posted by Michael Segel <mi...@hotmail.com>.
Since Tom isn't technical... ;-)
The short answer is No.
HBase is not capable of being a transactional because it doesn't support transactions.
Nor is HBase ACID compliant.
Having said that, yes you can use HBase to serve data in real time.
HTH
-Mike
> Subject: Re: Using HBase for real time transaction
> From: jignesh@websoft.com
> Date: Tue, 20 Sep 2011 17:25:17 -0400
> To: common-user@hadoop.apache.org
>
> Tom,
> Let me reword: can HBase be used as a transactional database(i.e. in replacement of mysql)?
>
> The requirement is to have real time read and write operations. I mean as soon as data is written the user should see the data(Here data should be written in Hbase).
>
> -Jignesh
>
>
> On Sep 20, 2011, at 5:11 PM, Tom Deutsch wrote:
>
> > Real-time means different things to different people. Can you share your
> > latency requirements from the time the data is generated to when it needs
> > to be consumed, or how you are thinking of using Hbase in the overall
> > flow?
> >
> > ------------------------------------------------
> > Tom Deutsch
> > Program Director
> > CTO Office: Information Management
> > Hadoop Product Manager / Customer Exec
> > IBM
> > 3565 Harbor Blvd
> > Costa Mesa, CA 92626-1420
> > tdeutsch@us.ibm.com
> >
> >
> >
> >
> > Jignesh Patel <ji...@websoft.com>
> > 09/20/2011 12:57 PM
> > Please respond to
> > common-user@hadoop.apache.org
> >
> >
> > To
> > common-user@hadoop.apache.org
> > cc
> >
> > Subject
> > Using HBase for real time transaction
> >
> >
> >
> >
> >
> >
> > We are exploring possibility of using HBase for the real time
> > transactions. Is that possible?
> >
> > -Jignesh
> >
>
Re: Using HBase for real time transaction
Posted by Jignesh Patel <ji...@websoft.com>.
Tom,
Let me reword: can HBase be used as a transactional database(i.e. in replacement of mysql)?
The requirement is to have real time read and write operations. I mean as soon as data is written the user should see the data(Here data should be written in Hbase).
-Jignesh
On Sep 20, 2011, at 5:11 PM, Tom Deutsch wrote:
> Real-time means different things to different people. Can you share your
> latency requirements from the time the data is generated to when it needs
> to be consumed, or how you are thinking of using Hbase in the overall
> flow?
>
> ------------------------------------------------
> Tom Deutsch
> Program Director
> CTO Office: Information Management
> Hadoop Product Manager / Customer Exec
> IBM
> 3565 Harbor Blvd
> Costa Mesa, CA 92626-1420
> tdeutsch@us.ibm.com
>
>
>
>
> Jignesh Patel <ji...@websoft.com>
> 09/20/2011 12:57 PM
> Please respond to
> common-user@hadoop.apache.org
>
>
> To
> common-user@hadoop.apache.org
> cc
>
> Subject
> Using HBase for real time transaction
>
>
>
>
>
>
> We are exploring possibility of using HBase for the real time
> transactions. Is that possible?
>
> -Jignesh
>
Re: Using HBase for real time transaction
Posted by Tom Deutsch <td...@us.ibm.com>.
Real-time means different things to different people. Can you share your
latency requirements from the time the data is generated to when it needs
to be consumed, or how you are thinking of using Hbase in the overall
flow?
------------------------------------------------
Tom Deutsch
Program Director
CTO Office: Information Management
Hadoop Product Manager / Customer Exec
IBM
3565 Harbor Blvd
Costa Mesa, CA 92626-1420
tdeutsch@us.ibm.com
Jignesh Patel <ji...@websoft.com>
09/20/2011 12:57 PM
Please respond to
common-user@hadoop.apache.org
To
common-user@hadoop.apache.org
cc
Subject
Using HBase for real time transaction
We are exploring possibility of using HBase for the real time
transactions. Is that possible?
-Jignesh