You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Imran M Yousuf <im...@gmail.com> on 2010/05/17 17:13:33 UTC
Availability Transaction and data integrity
Hi,
Currently we are designing an architecture for a Accounting SaaS and
e-commerce website. As both of them will store financial data -
transaction, redundancy, HA and data integrity is very important. As I
am not a master of HBase architecture and implementation I am eagerly
waiting for your comments on as follows:
* We will go live from January 2011, in that time frame should we
develop using 0.21-SNAPSHOT or should we stick to 0.20.x? Ideally I
would not want to go ahead with a snapshot in production and also
would not want to make an upgrade within few months (because of some
problems noticed in the mailing list regarding upgrade and I am a bit
skeptical about it in general).
* Transaction was a contrib module of HBase but it seems recently
removed from the 0.21-SNAPSHOT. In light of it what would be the way
to achieve transaction?
* NN was (if I am not mistaken) a SPoF, I also learnt that its
supposed to be fixed in 0.21, is that in trunk already?
* What kind of data loss should we design to?
* Is there any professional service provider who could help us train
for deployment, help optimize and in case we need emergency provide
service? (P.S. I contacted Cloudera via email 11 days back and still
waiting for a reply, may be they are not interested any alternate
would do great!)
I eagerly hope for some help and guideline on these queries.
--
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
Re: Availability Transaction and data integrity
Posted by Imran M Yousuf <im...@gmail.com>.
Ah ok. Thanks for clearing it JG.
Regards,
Imran
On Mon, May 17, 2010 at 11:10 PM, Jonathan Gray <jg...@facebook.com> wrote:
> What you read from the slides was about removing HBase's SPOF, not Hadoop's.
>
> From 0.20, HBase doesn't have a SPOF in that you can have backup masters. In more recent testing we've shown that while we support backup masters, there are still circumstances where the second master won't know what to do and your cluster could be left in an inconsistent state.
>
> That is being dealt with in 0.21 over in HBASE-2485.
>
> JG
>
>> -----Original Message-----
>> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
>> Sent: Monday, May 17, 2010 10:01 AM
>> To: hbase-user@hadoop.apache.org
>> Subject: Re: Availability Transaction and data integrity
>>
>> Thanks, my answers are inline too.
>>
>> On Mon, May 17, 2010 at 9:50 PM, Jonathan Gray <jg...@facebook.com>
>> wrote:
>> > Answers inline.
>> >
>> >> <snip />
>> >> * We will go live from January 2011, in that time frame should we
>> >> develop using 0.21-SNAPSHOT or should we stick to 0.20.x? Ideally I
>> >> would not want to go ahead with a snapshot in production and also
>> >> would not want to make an upgrade within few months (because of some
>> >> problems noticed in the mailing list regarding upgrade and I am a
>> bit
>> >> skeptical about it in general).
>> >
>> > If you need data durability (no data loss under node failure) then
>> you have no choice but to go with 0.21 once it is released. This is
>> not supported on the 0.20 line.
>> >
>> > There are a number of organizations who will be going live into
>> production on 0.21 in Q3 2010. You can be sure that there will be a
>> very well tested and stable 0.21 release by January 2011.
>> >
>>
>> Extremely great! Then we are on the right track working with 0.21
>>
>> >> * Transaction was a contrib module of HBase but it seems recently
>> >> removed from the 0.21-SNAPSHOT. In light of it what would be the way
>> >> to achieve transaction?
>> >
>> > It is still available but is being moved to GitHub. You can still
>> use it, it has just been moved out of the core code.
>> >
>> >> * NN was (if I am not mistaken) a SPoF, I also learnt that its
>> >> supposed to be fixed in 0.21, is that in trunk already?
>> >
>> > I'm not sure where you heard this was fixed in 0.21 as my
>> understanding is that it is not fixed in 0.21.
>> >
>> > There is work being done at Facebook (and I believe parallel work
>> being done elsewhere) to add a true backup NameNode. Once stabilized
>> this will be released and available to the public though it may not be
>> put into an official Hadoop release in an 0.21 timeframe.
>> >
>>
>> I see, in that case I misunderstood the statement from Slide #10 of
>> "ApacheCon2009: Practical HBase" - 'Removed SPoF, multi-master w/
>> automatic failover (ZK)'. What was this statement referring to as
>> master?
>>
>> Great to learn that its being worked on..
>>
>> >> * What kind of data loss should we design to?
>> >
>> > On 0.21, you should not have data loss. We are doing a lot of
>> testing on this to ensure stability and durability.
>> >
>>
>> This is extremely good news, it seems both my primary concerns are
>> getting addressed!
>>
>> >> * Is there any professional service provider who could help us train
>> >> for deployment, help optimize and in case we need emergency provide
>> >> service? (P.S. I contacted Cloudera via email 11 days back and still
>> >> waiting for a reply, may be they are not interested any alternate
>> >> would do great!)
>> >
>> > Cloudera should be able to help. They're active on this list but
>> perhaps don't want to use this forum to sell services. I would ping
>> them again off the list.
>> >
>>
>> Hmm, ok will try again with them.
>>
>> Thanks a lot!
>>
>> Imran
>>
>> >>
>> >> I eagerly hope for some help and guideline on these queries.
>> >>
>> >> --
>> >> Imran M Yousuf
>> >> Entrepreneur & Software Engineer
>> >> Smart IT Engineering
>> >> Dhaka, Bangladesh
>> >> Email: imran@smartitengineering.com
>> >> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>> >> Mobile: +880-1711402557
>> >
>>
>>
>>
>> --
>> Imran M Yousuf
>> Entrepreneur & Software Engineer
>> Smart IT Engineering
>> Dhaka, Bangladesh
>> Email: imran@smartitengineering.com
>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>> Mobile: +880-1711402557
>
--
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
RE: Availability Transaction and data integrity
Posted by Jonathan Gray <jg...@facebook.com>.
What you read from the slides was about removing HBase's SPOF, not Hadoop's.
>From 0.20, HBase doesn't have a SPOF in that you can have backup masters. In more recent testing we've shown that while we support backup masters, there are still circumstances where the second master won't know what to do and your cluster could be left in an inconsistent state.
That is being dealt with in 0.21 over in HBASE-2485.
JG
> -----Original Message-----
> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
> Sent: Monday, May 17, 2010 10:01 AM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Availability Transaction and data integrity
>
> Thanks, my answers are inline too.
>
> On Mon, May 17, 2010 at 9:50 PM, Jonathan Gray <jg...@facebook.com>
> wrote:
> > Answers inline.
> >
> >> <snip />
> >> * We will go live from January 2011, in that time frame should we
> >> develop using 0.21-SNAPSHOT or should we stick to 0.20.x? Ideally I
> >> would not want to go ahead with a snapshot in production and also
> >> would not want to make an upgrade within few months (because of some
> >> problems noticed in the mailing list regarding upgrade and I am a
> bit
> >> skeptical about it in general).
> >
> > If you need data durability (no data loss under node failure) then
> you have no choice but to go with 0.21 once it is released. This is
> not supported on the 0.20 line.
> >
> > There are a number of organizations who will be going live into
> production on 0.21 in Q3 2010. You can be sure that there will be a
> very well tested and stable 0.21 release by January 2011.
> >
>
> Extremely great! Then we are on the right track working with 0.21
>
> >> * Transaction was a contrib module of HBase but it seems recently
> >> removed from the 0.21-SNAPSHOT. In light of it what would be the way
> >> to achieve transaction?
> >
> > It is still available but is being moved to GitHub. You can still
> use it, it has just been moved out of the core code.
> >
> >> * NN was (if I am not mistaken) a SPoF, I also learnt that its
> >> supposed to be fixed in 0.21, is that in trunk already?
> >
> > I'm not sure where you heard this was fixed in 0.21 as my
> understanding is that it is not fixed in 0.21.
> >
> > There is work being done at Facebook (and I believe parallel work
> being done elsewhere) to add a true backup NameNode. Once stabilized
> this will be released and available to the public though it may not be
> put into an official Hadoop release in an 0.21 timeframe.
> >
>
> I see, in that case I misunderstood the statement from Slide #10 of
> "ApacheCon2009: Practical HBase" - 'Removed SPoF, multi-master w/
> automatic failover (ZK)'. What was this statement referring to as
> master?
>
> Great to learn that its being worked on..
>
> >> * What kind of data loss should we design to?
> >
> > On 0.21, you should not have data loss. We are doing a lot of
> testing on this to ensure stability and durability.
> >
>
> This is extremely good news, it seems both my primary concerns are
> getting addressed!
>
> >> * Is there any professional service provider who could help us train
> >> for deployment, help optimize and in case we need emergency provide
> >> service? (P.S. I contacted Cloudera via email 11 days back and still
> >> waiting for a reply, may be they are not interested any alternate
> >> would do great!)
> >
> > Cloudera should be able to help. They're active on this list but
> perhaps don't want to use this forum to sell services. I would ping
> them again off the list.
> >
>
> Hmm, ok will try again with them.
>
> Thanks a lot!
>
> Imran
>
> >>
> >> I eagerly hope for some help and guideline on these queries.
> >>
> >> --
> >> Imran M Yousuf
> >> Entrepreneur & Software Engineer
> >> Smart IT Engineering
> >> Dhaka, Bangladesh
> >> Email: imran@smartitengineering.com
> >> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> >> Mobile: +880-1711402557
> >
>
>
>
> --
> Imran M Yousuf
> Entrepreneur & Software Engineer
> Smart IT Engineering
> Dhaka, Bangladesh
> Email: imran@smartitengineering.com
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> Mobile: +880-1711402557
Re: Availability Transaction and data integrity
Posted by Imran M Yousuf <im...@gmail.com>.
Thanks, my answers are inline too.
On Mon, May 17, 2010 at 9:50 PM, Jonathan Gray <jg...@facebook.com> wrote:
> Answers inline.
>
>> <snip />
>> * We will go live from January 2011, in that time frame should we
>> develop using 0.21-SNAPSHOT or should we stick to 0.20.x? Ideally I
>> would not want to go ahead with a snapshot in production and also
>> would not want to make an upgrade within few months (because of some
>> problems noticed in the mailing list regarding upgrade and I am a bit
>> skeptical about it in general).
>
> If you need data durability (no data loss under node failure) then you have no choice but to go with 0.21 once it is released. This is not supported on the 0.20 line.
>
> There are a number of organizations who will be going live into production on 0.21 in Q3 2010. You can be sure that there will be a very well tested and stable 0.21 release by January 2011.
>
Extremely great! Then we are on the right track working with 0.21
>> * Transaction was a contrib module of HBase but it seems recently
>> removed from the 0.21-SNAPSHOT. In light of it what would be the way
>> to achieve transaction?
>
> It is still available but is being moved to GitHub. You can still use it, it has just been moved out of the core code.
>
>> * NN was (if I am not mistaken) a SPoF, I also learnt that its
>> supposed to be fixed in 0.21, is that in trunk already?
>
> I'm not sure where you heard this was fixed in 0.21 as my understanding is that it is not fixed in 0.21.
>
> There is work being done at Facebook (and I believe parallel work being done elsewhere) to add a true backup NameNode. Once stabilized this will be released and available to the public though it may not be put into an official Hadoop release in an 0.21 timeframe.
>
I see, in that case I misunderstood the statement from Slide #10 of
"ApacheCon2009: Practical HBase" - 'Removed SPoF, multi-master w/
automatic failover (ZK)'. What was this statement referring to as
master?
Great to learn that its being worked on..
>> * What kind of data loss should we design to?
>
> On 0.21, you should not have data loss. We are doing a lot of testing on this to ensure stability and durability.
>
This is extremely good news, it seems both my primary concerns are
getting addressed!
>> * Is there any professional service provider who could help us train
>> for deployment, help optimize and in case we need emergency provide
>> service? (P.S. I contacted Cloudera via email 11 days back and still
>> waiting for a reply, may be they are not interested any alternate
>> would do great!)
>
> Cloudera should be able to help. They're active on this list but perhaps don't want to use this forum to sell services. I would ping them again off the list.
>
Hmm, ok will try again with them.
Thanks a lot!
Imran
>>
>> I eagerly hope for some help and guideline on these queries.
>>
>> --
>> Imran M Yousuf
>> Entrepreneur & Software Engineer
>> Smart IT Engineering
>> Dhaka, Bangladesh
>> Email: imran@smartitengineering.com
>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>> Mobile: +880-1711402557
>
--
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
RE: Availability Transaction and data integrity
Posted by Jonathan Gray <jg...@facebook.com>.
Answers inline.
> -----Original Message-----
> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
> Sent: Monday, May 17, 2010 8:14 AM
> To: hbase-user@hadoop.apache.org
> Subject: Availability Transaction and data integrity
>
> Hi,
>
> Currently we are designing an architecture for a Accounting SaaS and
> e-commerce website. As both of them will store financial data -
> transaction, redundancy, HA and data integrity is very important. As I
> am not a master of HBase architecture and implementation I am eagerly
> waiting for your comments on as follows:
>
> * We will go live from January 2011, in that time frame should we
> develop using 0.21-SNAPSHOT or should we stick to 0.20.x? Ideally I
> would not want to go ahead with a snapshot in production and also
> would not want to make an upgrade within few months (because of some
> problems noticed in the mailing list regarding upgrade and I am a bit
> skeptical about it in general).
If you need data durability (no data loss under node failure) then you have no choice but to go with 0.21 once it is released. This is not supported on the 0.20 line.
There are a number of organizations who will be going live into production on 0.21 in Q3 2010. You can be sure that there will be a very well tested and stable 0.21 release by January 2011.
> * Transaction was a contrib module of HBase but it seems recently
> removed from the 0.21-SNAPSHOT. In light of it what would be the way
> to achieve transaction?
It is still available but is being moved to GitHub. You can still use it, it has just been moved out of the core code.
> * NN was (if I am not mistaken) a SPoF, I also learnt that its
> supposed to be fixed in 0.21, is that in trunk already?
I'm not sure where you heard this was fixed in 0.21 as my understanding is that it is not fixed in 0.21.
There is work being done at Facebook (and I believe parallel work being done elsewhere) to add a true backup NameNode. Once stabilized this will be released and available to the public though it may not be put into an official Hadoop release in an 0.21 timeframe.
> * What kind of data loss should we design to?
On 0.21, you should not have data loss. We are doing a lot of testing on this to ensure stability and durability.
> * Is there any professional service provider who could help us train
> for deployment, help optimize and in case we need emergency provide
> service? (P.S. I contacted Cloudera via email 11 days back and still
> waiting for a reply, may be they are not interested any alternate
> would do great!)
Cloudera should be able to help. They're active on this list but perhaps don't want to use this forum to sell services. I would ping them again off the list.
>
> I eagerly hope for some help and guideline on these queries.
>
> --
> Imran M Yousuf
> Entrepreneur & Software Engineer
> Smart IT Engineering
> Dhaka, Bangladesh
> Email: imran@smartitengineering.com
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> Mobile: +880-1711402557