You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Mark Laffoon <ml...@semanticresearch.com> on 2010/09/28 16:59:53 UTC
Upgrading 0.20.6 -> 0.89
We're using 0.20.6; we have a non-trivial application using many aspects
of hbase; we have a couple of customers in production; we understand this
is still pre-release, however we don't want to lose any data.
Will upgrading to 0.89 be a PITA?
Should we expect to be able to upgrade the servers without losing data?
Will there be tons of client code changes?
What about configuration changes (especially little changes that will bite
us)?
Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
If we do upgrade, what is the recommended package to get it from?
Thanks in advance for any or all answers,
Mark
Re: Upgrading 0.20.6 -> 0.89
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Sep 28, 2010 at 10:09 AM, Andrew Purtell <ap...@apache.org>wrote:
> A working equivalent of sync() in HDFS, and support for it.
>
> See http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3-b2-hbase/ ,
> especially: "HDFS improvements for HBase – along with the HDFS team at
> Facebook, we have contributed a number of important bug fixes and
> improvements for HDFS specifically to enable and improve HBase. These fixes
> include a working sync feature that allows full durability of every HBase
> edit, plus numerous performance and reliability fixes."
>
> You could also use the 0.20-append branch of ASF hadoop-common:
>
> http://github.com/apache/hadoop-common/tree/branch-0.20-append
>
>
FYI the branch-0.20-append is missing several patches that are in CDH3b2 to
fix up append. I don't know of anyone actually using branch-0.20-append, so
it doesn't get the testing that CDH does.
but CDH3 offers other benefits, check out their blog.
>
> Best regards,
>
> - Andy
>
>
> --- On Tue, 9/28/10, Renato Marroquín Mogrovejo <
> renatoj.marroquin@gmail.com> wrote:
>
> > From: Renato Marroquín Mogrovejo <re...@gmail.com>
> > Subject: Re: Upgrading 0.20.6 -> 0.89
> > To: user@hbase.apache.org
> > Date: Tuesday, September 28, 2010, 10:02 AM
> >
> > Just a quick question that often intrigues me, why do you guys
> > prefer the CDH3b2? and not a regular hadoop-0.20.X???
> > Thanks in advanced.
> >
> >
> > Renato M.
>
>
>
>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Upgrading 0.20.6 -> 0.89
Posted by Renato Marroquín Mogrovejo <re...@gmail.com>.
Thanks!
Renato M.
2010/9/28 Andrew Purtell <ap...@apache.org>
> A working equivalent of sync() in HDFS, and support for it.
>
> See http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3-b2-hbase/ ,
> especially: "HDFS improvements for HBase – along with the HDFS team at
> Facebook, we have contributed a number of important bug fixes and
> improvements for HDFS specifically to enable and improve HBase. These fixes
> include a working sync feature that allows full durability of every HBase
> edit, plus numerous performance and reliability fixes."
>
> You could also use the 0.20-append branch of ASF hadoop-common:
>
> http://github.com/apache/hadoop-common/tree/branch-0.20-append
>
> but CDH3 offers other benefits, check out their blog.
>
> Best regards,
>
> - Andy
>
>
> --- On Tue, 9/28/10, Renato Marroquín Mogrovejo <
> renatoj.marroquin@gmail.com> wrote:
>
> > From: Renato Marroquín Mogrovejo <re...@gmail.com>
> > Subject: Re: Upgrading 0.20.6 -> 0.89
> > To: user@hbase.apache.org
> > Date: Tuesday, September 28, 2010, 10:02 AM
> >
> > Just a quick question that often intrigues me, why do you guys
> > prefer the CDH3b2? and not a regular hadoop-0.20.X???
> > Thanks in advanced.
> >
> >
> > Renato M.
>
>
>
>
>
>
Re: Upgrading 0.20.6 -> 0.89
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Somes patches that improve throughput for HBase, although you also
need a HBase-side patch (HBASE-2467). They also backported stuff from
0.21 that's never going to be in 0.20-append. That's our main reasons
to use CDh3b2.
J-D
On Tue, Sep 28, 2010 at 10:02 AM, Renato Marroquín Mogrovejo
<re...@gmail.com> wrote:
> Just a quick question that often intrigues me, why do you guys prefer the
> CDH3b2? and not a regular hadoop-0.20.X???
> Thanks in advanced.
>
>
> Renato M.
>
> 2010/9/28 Jean-Daniel Cryans <jd...@apache.org>
>
>> > Will upgrading to 0.89 be a PITA?
>>
>> Unless you still use the deprecated APIs, it's actually just a matter
>> of replacing the distribution and restarting.
>>
>> >
>> > Should we expect to be able to upgrade the servers without losing data?
>>
>> Definitely, since no upgrade of the filesystem format is required. But
>> it's always a good practice to backup your data before any upgrade.
>>
>> >
>> > Will there be tons of client code changes?
>>
>> See first answer.
>>
>> >
>> > What about configuration changes (especially little changes that will
>> bite
>> > us)?
>>
>> When we upgraded we only added dfs.support.append set to true.
>>
>> >
>> > Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
>>
>> If you want data durability (eg no data loss), yes.
>>
>> >
>> > If we do upgrade, what is the recommended package to get it from?
>>
>> We use CDH3b2's hadoop, I'd recommend that, but you can also use the
>> head of the Hadoop 0.20-append branch. We have our own HBase "distro"
>> that we publish here http://github.com/stumbleupon/hbase, this is
>> what's currently in production. It's not much different from
>> 0.89.20100924 (which is still RC1), mainly fixes and improvements for
>> cluster replication that will eventually make it to core hbase, and
>> some neat changes to the ThriftServer to enable async ICVs.
>>
>> J-D
>>
>
Re: Upgrading 0.20.6 -> 0.89
Posted by Andrew Purtell <ap...@apache.org>.
A working equivalent of sync() in HDFS, and support for it.
See http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3-b2-hbase/ , especially: "HDFS improvements for HBase – along with the HDFS team at Facebook, we have contributed a number of important bug fixes and improvements for HDFS specifically to enable and improve HBase. These fixes include a working sync feature that allows full durability of every HBase edit, plus numerous performance and reliability fixes."
You could also use the 0.20-append branch of ASF hadoop-common:
http://github.com/apache/hadoop-common/tree/branch-0.20-append
but CDH3 offers other benefits, check out their blog.
Best regards,
- Andy
--- On Tue, 9/28/10, Renato Marroquín Mogrovejo <re...@gmail.com> wrote:
> From: Renato Marroquín Mogrovejo <re...@gmail.com>
> Subject: Re: Upgrading 0.20.6 -> 0.89
> To: user@hbase.apache.org
> Date: Tuesday, September 28, 2010, 10:02 AM
>
> Just a quick question that often intrigues me, why do you guys
> prefer the CDH3b2? and not a regular hadoop-0.20.X???
> Thanks in advanced.
>
>
> Renato M.
Re: Upgrading 0.20.6 -> 0.89
Posted by Renato Marroquín Mogrovejo <re...@gmail.com>.
Just a quick question that often intrigues me, why do you guys prefer the
CDH3b2? and not a regular hadoop-0.20.X???
Thanks in advanced.
Renato M.
2010/9/28 Jean-Daniel Cryans <jd...@apache.org>
> > Will upgrading to 0.89 be a PITA?
>
> Unless you still use the deprecated APIs, it's actually just a matter
> of replacing the distribution and restarting.
>
> >
> > Should we expect to be able to upgrade the servers without losing data?
>
> Definitely, since no upgrade of the filesystem format is required. But
> it's always a good practice to backup your data before any upgrade.
>
> >
> > Will there be tons of client code changes?
>
> See first answer.
>
> >
> > What about configuration changes (especially little changes that will
> bite
> > us)?
>
> When we upgraded we only added dfs.support.append set to true.
>
> >
> > Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
>
> If you want data durability (eg no data loss), yes.
>
> >
> > If we do upgrade, what is the recommended package to get it from?
>
> We use CDH3b2's hadoop, I'd recommend that, but you can also use the
> head of the Hadoop 0.20-append branch. We have our own HBase "distro"
> that we publish here http://github.com/stumbleupon/hbase, this is
> what's currently in production. It's not much different from
> 0.89.20100924 (which is still RC1), mainly fixes and improvements for
> cluster replication that will eventually make it to core hbase, and
> some neat changes to the ThriftServer to enable async ICVs.
>
> J-D
>
Re: Upgrading 0.20.6 -> 0.89
Posted by Jean-Daniel Cryans <jd...@apache.org>.
> Will upgrading to 0.89 be a PITA?
Unless you still use the deprecated APIs, it's actually just a matter
of replacing the distribution and restarting.
>
> Should we expect to be able to upgrade the servers without losing data?
Definitely, since no upgrade of the filesystem format is required. But
it's always a good practice to backup your data before any upgrade.
>
> Will there be tons of client code changes?
See first answer.
>
> What about configuration changes (especially little changes that will bite
> us)?
When we upgraded we only added dfs.support.append set to true.
>
> Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
If you want data durability (eg no data loss), yes.
>
> If we do upgrade, what is the recommended package to get it from?
We use CDH3b2's hadoop, I'd recommend that, but you can also use the
head of the Hadoop 0.20-append branch. We have our own HBase "distro"
that we publish here http://github.com/stumbleupon/hbase, this is
what's currently in production. It's not much different from
0.89.20100924 (which is still RC1), mainly fixes and improvements for
cluster replication that will eventually make it to core hbase, and
some neat changes to the ThriftServer to enable async ICVs.
J-D
Re: Upgrading 0.20.6 -> 0.89
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Ok, thank you, Ryan. I certainly can't claim a deep intrinsic knowledge on
GC, just read the Sun's GC guide which doesn't seem to draw that clear
distinction between full gc and tenured gen GC, and since then acted on
their examples given for figuring pause times -- which seem to refer to
tenured gen gc only, i guess.
On Wed, Sep 29, 2010 at 1:54 PM, Ryan Rawson <ry...@gmail.com> wrote:
> FullGC is not the CMS cycle... During a FullGC the entire main heap is
> rewritten and compacted. This can happen due to fragmentation issues,
> and can happen if the CMS cycle does not finish releasing enough
> memory before the minor GC needs to promote (aka: concurrent promotion
> failure, or something to that effect). This is NOT the same as the CMS
> cycle, and during FullGC (rewriting/compacting heap) all threads are
> indeed paused.
>
> On a modern i7 oriented architecture with the fast memory buses I have
> heard people say that it is about 1 second per GB of heap. 8 gb heap
> = 8s pause.
>
> However experience has shown that this can be substantially longer, up
> to 20 seconds, sometimes 2 x 20 second pauses back to back causing ZK
> to time out and the master to presume the RS is dead and recover the
> log. At this point when the RS recovers it must self terminate
> because the regions dont 'belong' to itself anymore.
>
>
>
> On Wed, Sep 29, 2010 at 10:51 AM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> >> Full GCs do happen. We have it at 40 seconds here.
> >
> > Jean-Daniel, Is it total with concurrent CMS?
> >
> > 40 seconds is a plausible number for full CMS, even more plausible for
> > i-CMS, so i assume that's what you are quoting here.
> >
> > But CMS doesn't pause jvm for that much time. Most of that time is spent
> in
> > a single background thread. It only pauses the entire jvm for
> non-concurrent
> > steps that are supposed to be optimized to be very short. In my
> experience
> > non-concurrent pauses are normally under 200ms even for processes with a
> > very heavy memory use (~16G heap) (unless you are having a really bad
> case
> > of GC thrashing). I never saw one actually logging any more than
> 200-300ms
> > for non-conrrent remark phase -- and that was a really bad case. I never
> > checked HBase's GC logs so far though. If Hbase does have those
> > non-concurrent pauses any bigger, >1 s, that's kind of .. unusual and
> > severe, for lack of a better word, for a real time process, imo.
> >
> > So therefore the process can't go 'numb' for that much from ZK's point of
> > view, if it does, then it's a problem imo. It's more likely that such a
> > delay may be associated with VM swap activities, but if GC can freeze
> HBase
> > that bad.. that would be bad indeed from acceptance point of view.
> >
> >
> > On Wed, Sep 29, 2010 at 9:37 AM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
> >
> >> I'd say it mostly depends on your tolerance to regions being
> >> unavailable while the recovery happens. You have to account for the ZK
> >> timeout (60 secs by default), plus the time to split (I don't have any
> >> good metric for that, usually it's kinda fast but you should try it
> >> with your data), plus the time to reassign the regions and, when they
> >> open, the replay of the split logs. Some tips:
> >>
> >> - Lower ZK timeout means you are more likely to have long GC pauses
> >> taking down your region server (well, the master assumes the RS is
> >> dead). Full GCs do happen. We have it at 40 seconds here.
> >> - Smaller HLogs / fewer HLogs means you will force flush regions a
> >> lot more in order to keep the data to be replayed lower. Here we have
> >> hbase.regionserver.maxlogs=8 so we do flush a lot, but we serve a live
> >> website. In the future, distributed log splitting (once committed)
> >> will make it faster.
> >>
> >> J-D
> >>
> >> On Wed, Sep 29, 2010 at 6:12 AM, Daniel Einspanjer
> >> <de...@mozilla.com> wrote:
> >> > Question regarding configuration and tuning...
> >> >
> >> > Our current configuration/schema has fairly low hlog rollover sizes to
> >> keep
> >> > the possibility of data loss to a minimum. When we upgrade to .89
> with
> >> > append support, I imagine we'll be able to safely set this to a much
> >> larger
> >> > size. Are there any rough guidelines for what a good values should be
> >> now?
> >> >
> >> > -Daniel
> >> >
> >> > On 9/28/10 6:13 PM, Buttler, David wrote:
> >> >>
> >> >> Fantastic news, I look forward to it
> >> >> Dave
> >> >>
> >> >> -----Original Message-----
> >> >> From: Todd Lipcon [mailto:todd@cloudera.com]
> >> >> Sent: Tuesday, September 28, 2010 11:25 AM
> >> >> To: user@hbase.apache.org
> >> >> Subject: Re: Upgrading 0.20.6 -> 0.89
> >> >>
> >> >> On Tue, Sep 28, 2010 at 9:35 AM, Buttler, David<bu...@llnl.gov>
> >> wrote:
> >> >>
> >> >>> I currently suggest that you use the CDH3 hadoop package.
> Apparently
> >> >>> StumbleUpon has a production version of 0.89 that they are using.
> It
> >> >>> would
> >> >>> be helpful if Cloudera put that in their distribution.
> >> >>>
> >> >>>
> >> >> Working on it ;-) CDH3b3 should be available in about 2 weeks and
> will
> >> >> include an HBase version that's very similar to what StumbleUpon has
> >> >> published.
> >> >>
> >> >> -Todd
> >> >>
> >> >>>
> >> >>> -----Original Message-----
> >> >>> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> >> >>> Sent: Tuesday, September 28, 2010 8:00 AM
> >> >>> To: user@hbase.apache.org
> >> >>> Subject: Upgrading 0.20.6 -> 0.89
> >> >>>
> >> >>> We're using 0.20.6; we have a non-trivial application using many
> >> aspects
> >> >>> of hbase; we have a couple of customers in production; we understand
> >> this
> >> >>> is still pre-release, however we don't want to lose any data.
> >> >>>
> >> >>>
> >> >>>
> >> >>> Will upgrading to 0.89 be a PITA?
> >> >>>
> >> >>> Should we expect to be able to upgrade the servers without losing
> data?
> >> >>>
> >> >>> Will there be tons of client code changes?
> >> >>>
> >> >>> What about configuration changes (especially little changes that
> will
> >> >>> bite
> >> >>> us)?
> >> >>>
> >> >>> Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
> >> >>>
> >> >>> If we do upgrade, what is the recommended package to get it from?
> >> >>>
> >> >>>
> >> >>>
> >> >>> Thanks in advance for any or all answers,
> >> >>>
> >> >>> Mark
> >> >>>
> >> >>>
> >> >>
> >> >
> >>
> >
>
Re: Upgrading 0.20.6 -> 0.89
Posted by Ryan Rawson <ry...@gmail.com>.
FullGC is not the CMS cycle... During a FullGC the entire main heap is
rewritten and compacted. This can happen due to fragmentation issues,
and can happen if the CMS cycle does not finish releasing enough
memory before the minor GC needs to promote (aka: concurrent promotion
failure, or something to that effect). This is NOT the same as the CMS
cycle, and during FullGC (rewriting/compacting heap) all threads are
indeed paused.
On a modern i7 oriented architecture with the fast memory buses I have
heard people say that it is about 1 second per GB of heap. 8 gb heap
= 8s pause.
However experience has shown that this can be substantially longer, up
to 20 seconds, sometimes 2 x 20 second pauses back to back causing ZK
to time out and the master to presume the RS is dead and recover the
log. At this point when the RS recovers it must self terminate
because the regions dont 'belong' to itself anymore.
On Wed, Sep 29, 2010 at 10:51 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
>> Full GCs do happen. We have it at 40 seconds here.
>
> Jean-Daniel, Is it total with concurrent CMS?
>
> 40 seconds is a plausible number for full CMS, even more plausible for
> i-CMS, so i assume that's what you are quoting here.
>
> But CMS doesn't pause jvm for that much time. Most of that time is spent in
> a single background thread. It only pauses the entire jvm for non-concurrent
> steps that are supposed to be optimized to be very short. In my experience
> non-concurrent pauses are normally under 200ms even for processes with a
> very heavy memory use (~16G heap) (unless you are having a really bad case
> of GC thrashing). I never saw one actually logging any more than 200-300ms
> for non-conrrent remark phase -- and that was a really bad case. I never
> checked HBase's GC logs so far though. If Hbase does have those
> non-concurrent pauses any bigger, >1 s, that's kind of .. unusual and
> severe, for lack of a better word, for a real time process, imo.
>
> So therefore the process can't go 'numb' for that much from ZK's point of
> view, if it does, then it's a problem imo. It's more likely that such a
> delay may be associated with VM swap activities, but if GC can freeze HBase
> that bad.. that would be bad indeed from acceptance point of view.
>
>
> On Wed, Sep 29, 2010 at 9:37 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
>> I'd say it mostly depends on your tolerance to regions being
>> unavailable while the recovery happens. You have to account for the ZK
>> timeout (60 secs by default), plus the time to split (I don't have any
>> good metric for that, usually it's kinda fast but you should try it
>> with your data), plus the time to reassign the regions and, when they
>> open, the replay of the split logs. Some tips:
>>
>> - Lower ZK timeout means you are more likely to have long GC pauses
>> taking down your region server (well, the master assumes the RS is
>> dead). Full GCs do happen. We have it at 40 seconds here.
>> - Smaller HLogs / fewer HLogs means you will force flush regions a
>> lot more in order to keep the data to be replayed lower. Here we have
>> hbase.regionserver.maxlogs=8 so we do flush a lot, but we serve a live
>> website. In the future, distributed log splitting (once committed)
>> will make it faster.
>>
>> J-D
>>
>> On Wed, Sep 29, 2010 at 6:12 AM, Daniel Einspanjer
>> <de...@mozilla.com> wrote:
>> > Question regarding configuration and tuning...
>> >
>> > Our current configuration/schema has fairly low hlog rollover sizes to
>> keep
>> > the possibility of data loss to a minimum. When we upgrade to .89 with
>> > append support, I imagine we'll be able to safely set this to a much
>> larger
>> > size. Are there any rough guidelines for what a good values should be
>> now?
>> >
>> > -Daniel
>> >
>> > On 9/28/10 6:13 PM, Buttler, David wrote:
>> >>
>> >> Fantastic news, I look forward to it
>> >> Dave
>> >>
>> >> -----Original Message-----
>> >> From: Todd Lipcon [mailto:todd@cloudera.com]
>> >> Sent: Tuesday, September 28, 2010 11:25 AM
>> >> To: user@hbase.apache.org
>> >> Subject: Re: Upgrading 0.20.6 -> 0.89
>> >>
>> >> On Tue, Sep 28, 2010 at 9:35 AM, Buttler, David<bu...@llnl.gov>
>> wrote:
>> >>
>> >>> I currently suggest that you use the CDH3 hadoop package. Apparently
>> >>> StumbleUpon has a production version of 0.89 that they are using. It
>> >>> would
>> >>> be helpful if Cloudera put that in their distribution.
>> >>>
>> >>>
>> >> Working on it ;-) CDH3b3 should be available in about 2 weeks and will
>> >> include an HBase version that's very similar to what StumbleUpon has
>> >> published.
>> >>
>> >> -Todd
>> >>
>> >>>
>> >>> -----Original Message-----
>> >>> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
>> >>> Sent: Tuesday, September 28, 2010 8:00 AM
>> >>> To: user@hbase.apache.org
>> >>> Subject: Upgrading 0.20.6 -> 0.89
>> >>>
>> >>> We're using 0.20.6; we have a non-trivial application using many
>> aspects
>> >>> of hbase; we have a couple of customers in production; we understand
>> this
>> >>> is still pre-release, however we don't want to lose any data.
>> >>>
>> >>>
>> >>>
>> >>> Will upgrading to 0.89 be a PITA?
>> >>>
>> >>> Should we expect to be able to upgrade the servers without losing data?
>> >>>
>> >>> Will there be tons of client code changes?
>> >>>
>> >>> What about configuration changes (especially little changes that will
>> >>> bite
>> >>> us)?
>> >>>
>> >>> Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
>> >>>
>> >>> If we do upgrade, what is the recommended package to get it from?
>> >>>
>> >>>
>> >>>
>> >>> Thanks in advance for any or all answers,
>> >>>
>> >>> Mark
>> >>>
>> >>>
>> >>
>> >
>>
>
Re: Upgrading 0.20.6 -> 0.89
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
> Full GCs do happen. We have it at 40 seconds here.
Jean-Daniel, Is it total with concurrent CMS?
40 seconds is a plausible number for full CMS, even more plausible for
i-CMS, so i assume that's what you are quoting here.
But CMS doesn't pause jvm for that much time. Most of that time is spent in
a single background thread. It only pauses the entire jvm for non-concurrent
steps that are supposed to be optimized to be very short. In my experience
non-concurrent pauses are normally under 200ms even for processes with a
very heavy memory use (~16G heap) (unless you are having a really bad case
of GC thrashing). I never saw one actually logging any more than 200-300ms
for non-conrrent remark phase -- and that was a really bad case. I never
checked HBase's GC logs so far though. If Hbase does have those
non-concurrent pauses any bigger, >1 s, that's kind of .. unusual and
severe, for lack of a better word, for a real time process, imo.
So therefore the process can't go 'numb' for that much from ZK's point of
view, if it does, then it's a problem imo. It's more likely that such a
delay may be associated with VM swap activities, but if GC can freeze HBase
that bad.. that would be bad indeed from acceptance point of view.
On Wed, Sep 29, 2010 at 9:37 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> I'd say it mostly depends on your tolerance to regions being
> unavailable while the recovery happens. You have to account for the ZK
> timeout (60 secs by default), plus the time to split (I don't have any
> good metric for that, usually it's kinda fast but you should try it
> with your data), plus the time to reassign the regions and, when they
> open, the replay of the split logs. Some tips:
>
> - Lower ZK timeout means you are more likely to have long GC pauses
> taking down your region server (well, the master assumes the RS is
> dead). Full GCs do happen. We have it at 40 seconds here.
> - Smaller HLogs / fewer HLogs means you will force flush regions a
> lot more in order to keep the data to be replayed lower. Here we have
> hbase.regionserver.maxlogs=8 so we do flush a lot, but we serve a live
> website. In the future, distributed log splitting (once committed)
> will make it faster.
>
> J-D
>
> On Wed, Sep 29, 2010 at 6:12 AM, Daniel Einspanjer
> <de...@mozilla.com> wrote:
> > Question regarding configuration and tuning...
> >
> > Our current configuration/schema has fairly low hlog rollover sizes to
> keep
> > the possibility of data loss to a minimum. When we upgrade to .89 with
> > append support, I imagine we'll be able to safely set this to a much
> larger
> > size. Are there any rough guidelines for what a good values should be
> now?
> >
> > -Daniel
> >
> > On 9/28/10 6:13 PM, Buttler, David wrote:
> >>
> >> Fantastic news, I look forward to it
> >> Dave
> >>
> >> -----Original Message-----
> >> From: Todd Lipcon [mailto:todd@cloudera.com]
> >> Sent: Tuesday, September 28, 2010 11:25 AM
> >> To: user@hbase.apache.org
> >> Subject: Re: Upgrading 0.20.6 -> 0.89
> >>
> >> On Tue, Sep 28, 2010 at 9:35 AM, Buttler, David<bu...@llnl.gov>
> wrote:
> >>
> >>> I currently suggest that you use the CDH3 hadoop package. Apparently
> >>> StumbleUpon has a production version of 0.89 that they are using. It
> >>> would
> >>> be helpful if Cloudera put that in their distribution.
> >>>
> >>>
> >> Working on it ;-) CDH3b3 should be available in about 2 weeks and will
> >> include an HBase version that's very similar to what StumbleUpon has
> >> published.
> >>
> >> -Todd
> >>
> >>>
> >>> -----Original Message-----
> >>> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> >>> Sent: Tuesday, September 28, 2010 8:00 AM
> >>> To: user@hbase.apache.org
> >>> Subject: Upgrading 0.20.6 -> 0.89
> >>>
> >>> We're using 0.20.6; we have a non-trivial application using many
> aspects
> >>> of hbase; we have a couple of customers in production; we understand
> this
> >>> is still pre-release, however we don't want to lose any data.
> >>>
> >>>
> >>>
> >>> Will upgrading to 0.89 be a PITA?
> >>>
> >>> Should we expect to be able to upgrade the servers without losing data?
> >>>
> >>> Will there be tons of client code changes?
> >>>
> >>> What about configuration changes (especially little changes that will
> >>> bite
> >>> us)?
> >>>
> >>> Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
> >>>
> >>> If we do upgrade, what is the recommended package to get it from?
> >>>
> >>>
> >>>
> >>> Thanks in advance for any or all answers,
> >>>
> >>> Mark
> >>>
> >>>
> >>
> >
>
Re: Upgrading 0.20.6 -> 0.89
Posted by Jean-Daniel Cryans <jd...@apache.org>.
I'd say it mostly depends on your tolerance to regions being
unavailable while the recovery happens. You have to account for the ZK
timeout (60 secs by default), plus the time to split (I don't have any
good metric for that, usually it's kinda fast but you should try it
with your data), plus the time to reassign the regions and, when they
open, the replay of the split logs. Some tips:
- Lower ZK timeout means you are more likely to have long GC pauses
taking down your region server (well, the master assumes the RS is
dead). Full GCs do happen. We have it at 40 seconds here.
- Smaller HLogs / fewer HLogs means you will force flush regions a
lot more in order to keep the data to be replayed lower. Here we have
hbase.regionserver.maxlogs=8 so we do flush a lot, but we serve a live
website. In the future, distributed log splitting (once committed)
will make it faster.
J-D
On Wed, Sep 29, 2010 at 6:12 AM, Daniel Einspanjer
<de...@mozilla.com> wrote:
> Question regarding configuration and tuning...
>
> Our current configuration/schema has fairly low hlog rollover sizes to keep
> the possibility of data loss to a minimum. When we upgrade to .89 with
> append support, I imagine we'll be able to safely set this to a much larger
> size. Are there any rough guidelines for what a good values should be now?
>
> -Daniel
>
> On 9/28/10 6:13 PM, Buttler, David wrote:
>>
>> Fantastic news, I look forward to it
>> Dave
>>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Tuesday, September 28, 2010 11:25 AM
>> To: user@hbase.apache.org
>> Subject: Re: Upgrading 0.20.6 -> 0.89
>>
>> On Tue, Sep 28, 2010 at 9:35 AM, Buttler, David<bu...@llnl.gov> wrote:
>>
>>> I currently suggest that you use the CDH3 hadoop package. Apparently
>>> StumbleUpon has a production version of 0.89 that they are using. It
>>> would
>>> be helpful if Cloudera put that in their distribution.
>>>
>>>
>> Working on it ;-) CDH3b3 should be available in about 2 weeks and will
>> include an HBase version that's very similar to what StumbleUpon has
>> published.
>>
>> -Todd
>>
>>>
>>> -----Original Message-----
>>> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
>>> Sent: Tuesday, September 28, 2010 8:00 AM
>>> To: user@hbase.apache.org
>>> Subject: Upgrading 0.20.6 -> 0.89
>>>
>>> We're using 0.20.6; we have a non-trivial application using many aspects
>>> of hbase; we have a couple of customers in production; we understand this
>>> is still pre-release, however we don't want to lose any data.
>>>
>>>
>>>
>>> Will upgrading to 0.89 be a PITA?
>>>
>>> Should we expect to be able to upgrade the servers without losing data?
>>>
>>> Will there be tons of client code changes?
>>>
>>> What about configuration changes (especially little changes that will
>>> bite
>>> us)?
>>>
>>> Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
>>>
>>> If we do upgrade, what is the recommended package to get it from?
>>>
>>>
>>>
>>> Thanks in advance for any or all answers,
>>>
>>> Mark
>>>
>>>
>>
>
Re: Upgrading 0.20.6 -> 0.89
Posted by Daniel Einspanjer <de...@mozilla.com>.
Question regarding configuration and tuning...
Our current configuration/schema has fairly low hlog rollover sizes to
keep the possibility of data loss to a minimum. When we upgrade to .89
with append support, I imagine we'll be able to safely set this to a
much larger size. Are there any rough guidelines for what a good values
should be now?
-Daniel
On 9/28/10 6:13 PM, Buttler, David wrote:
> Fantastic news, I look forward to it
> Dave
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Tuesday, September 28, 2010 11:25 AM
> To: user@hbase.apache.org
> Subject: Re: Upgrading 0.20.6 -> 0.89
>
> On Tue, Sep 28, 2010 at 9:35 AM, Buttler, David<bu...@llnl.gov> wrote:
>
>> I currently suggest that you use the CDH3 hadoop package. Apparently
>> StumbleUpon has a production version of 0.89 that they are using. It would
>> be helpful if Cloudera put that in their distribution.
>>
>>
> Working on it ;-) CDH3b3 should be available in about 2 weeks and will
> include an HBase version that's very similar to what StumbleUpon has
> published.
>
> -Todd
>
>>
>> -----Original Message-----
>> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
>> Sent: Tuesday, September 28, 2010 8:00 AM
>> To: user@hbase.apache.org
>> Subject: Upgrading 0.20.6 -> 0.89
>>
>> We're using 0.20.6; we have a non-trivial application using many aspects
>> of hbase; we have a couple of customers in production; we understand this
>> is still pre-release, however we don't want to lose any data.
>>
>>
>>
>> Will upgrading to 0.89 be a PITA?
>>
>> Should we expect to be able to upgrade the servers without losing data?
>>
>> Will there be tons of client code changes?
>>
>> What about configuration changes (especially little changes that will bite
>> us)?
>>
>> Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
>>
>> If we do upgrade, what is the recommended package to get it from?
>>
>>
>>
>> Thanks in advance for any or all answers,
>>
>> Mark
>>
>>
>
RE: Upgrading 0.20.6 -> 0.89
Posted by "Buttler, David" <bu...@llnl.gov>.
Fantastic news, I look forward to it
Dave
-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Tuesday, September 28, 2010 11:25 AM
To: user@hbase.apache.org
Subject: Re: Upgrading 0.20.6 -> 0.89
On Tue, Sep 28, 2010 at 9:35 AM, Buttler, David <bu...@llnl.gov> wrote:
>
> I currently suggest that you use the CDH3 hadoop package. Apparently
> StumbleUpon has a production version of 0.89 that they are using. It would
> be helpful if Cloudera put that in their distribution.
>
>
Working on it ;-) CDH3b3 should be available in about 2 weeks and will
include an HBase version that's very similar to what StumbleUpon has
published.
-Todd
>
>
> -----Original Message-----
> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> Sent: Tuesday, September 28, 2010 8:00 AM
> To: user@hbase.apache.org
> Subject: Upgrading 0.20.6 -> 0.89
>
> We're using 0.20.6; we have a non-trivial application using many aspects
> of hbase; we have a couple of customers in production; we understand this
> is still pre-release, however we don't want to lose any data.
>
>
>
> Will upgrading to 0.89 be a PITA?
>
> Should we expect to be able to upgrade the servers without losing data?
>
> Will there be tons of client code changes?
>
> What about configuration changes (especially little changes that will bite
> us)?
>
> Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
>
> If we do upgrade, what is the recommended package to get it from?
>
>
>
> Thanks in advance for any or all answers,
>
> Mark
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Upgrading 0.20.6 -> 0.89
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Sep 28, 2010 at 9:35 AM, Buttler, David <bu...@llnl.gov> wrote:
>
> I currently suggest that you use the CDH3 hadoop package. Apparently
> StumbleUpon has a production version of 0.89 that they are using. It would
> be helpful if Cloudera put that in their distribution.
>
>
Working on it ;-) CDH3b3 should be available in about 2 weeks and will
include an HBase version that's very similar to what StumbleUpon has
published.
-Todd
>
>
> -----Original Message-----
> From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
> Sent: Tuesday, September 28, 2010 8:00 AM
> To: user@hbase.apache.org
> Subject: Upgrading 0.20.6 -> 0.89
>
> We're using 0.20.6; we have a non-trivial application using many aspects
> of hbase; we have a couple of customers in production; we understand this
> is still pre-release, however we don't want to lose any data.
>
>
>
> Will upgrading to 0.89 be a PITA?
>
> Should we expect to be able to upgrade the servers without losing data?
>
> Will there be tons of client code changes?
>
> What about configuration changes (especially little changes that will bite
> us)?
>
> Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
>
> If we do upgrade, what is the recommended package to get it from?
>
>
>
> Thanks in advance for any or all answers,
>
> Mark
>
>
--
Todd Lipcon
Software Engineer, Cloudera
RE: Upgrading 0.20.6 -> 0.89
Posted by "Buttler, David" <bu...@llnl.gov>.
I have tried upgrading on one of my test clusters. I can say that the code changes are relatively small and minor. Things that I had to change:
1) how I was creating my Configuration objects -- using HBaseConfiguration.create() instead of new HBaseConfiguration()
2) how I was defining my column families in creating a table (since I wasn't just using the defaults)
I don't know about all of the little config changes that have bitten me and I haven't noticed yet.
I currently suggest that you use the CDH3 hadoop package. Apparently StumbleUpon has a production version of 0.89 that they are using. It would be helpful if Cloudera put that in their distribution.
Dave
-----Original Message-----
From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
Sent: Tuesday, September 28, 2010 8:00 AM
To: user@hbase.apache.org
Subject: Upgrading 0.20.6 -> 0.89
We're using 0.20.6; we have a non-trivial application using many aspects
of hbase; we have a couple of customers in production; we understand this
is still pre-release, however we don't want to lose any data.
Will upgrading to 0.89 be a PITA?
Should we expect to be able to upgrade the servers without losing data?
Will there be tons of client code changes?
What about configuration changes (especially little changes that will bite
us)?
Do we need/want to upgrade hadoop at all (we're on 0.20.2)?
If we do upgrade, what is the recommended package to get it from?
Thanks in advance for any or all answers,
Mark