You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by lars hofhansl <la...@apache.org> on 2014/12/15 19:53:07 UTC
0.94 going forward
Over the past few months the rate of the change into 0.94 has slowed significantly.
0.94.25 was released on Nov 15th, and since then we had only 4 changes.
This could mean two things: (1) 0.94 is very stable now or (2) nobody is using it (at least nobody is contributing to it anymore).
If anybody out there is still using 0.94 and is not planning to upgrade to 0.98 or later soon (which will required downtime), please speak up.
Otherwise it might be time to think about EOL'ing 0.94.
It's not actually much work to do these releases, especially when they are so small, but I'd like to continue only if they are actually used.
In any case, I am going to spin 0.94.26 with the current 4 fixes today or tomorrow.
-- Lars
Re: 0.94 going forward
Posted by lars hofhansl <la...@apache.org>.
Not from the source repository (we still have 0.90 and 0.92 there) and not necessarily from maven and the Apache archive. We would just stop the regular release cadence and not backport fixes automatically anymore.
Since this is open source everybody can propose a patch and if we'd get enough patches, presumably we'd do a new release.
-- Lars
From: Henry Saputra <he...@gmail.com>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Monday, December 15, 2014 4:28 PM
Subject: Re: 0.94 going forward
HI Lars,
When you said EOL 0.94 would that means that the artifact would be
removed from maven central and also the tag from source?
Thanks,
Henry
On Mon, Dec 15, 2014 at 10:53 AM, lars hofhansl <la...@apache.org> wrote:
> Over the past few months the rate of the change into 0.94 has slowed significantly.
> 0.94.25 was released on Nov 15th, and since then we had only 4 changes.
>
> This could mean two things: (1) 0.94 is very stable now or (2) nobody is using it (at least nobody is contributing to it anymore).
>
>
> If anybody out there is still using 0.94 and is not planning to upgrade to 0.98 or later soon (which will required downtime), please speak up.
> Otherwise it might be time to think about EOL'ing 0.94.
>
> It's not actually much work to do these releases, especially when they are so small, but I'd like to continue only if they are actually used.
> In any case, I am going to spin 0.94.26 with the current 4 fixes today or tomorrow.
>
> -- Lars
Re: 0.94 going forward
Posted by Henry Saputra <he...@gmail.com>.
HI Lars,
When you said EOL 0.94 would that means that the artifact would be
removed from maven central and also the tag from source?
Thanks,
Henry
On Mon, Dec 15, 2014 at 10:53 AM, lars hofhansl <la...@apache.org> wrote:
> Over the past few months the rate of the change into 0.94 has slowed significantly.
> 0.94.25 was released on Nov 15th, and since then we had only 4 changes.
>
> This could mean two things: (1) 0.94 is very stable now or (2) nobody is using it (at least nobody is contributing to it anymore).
>
>
> If anybody out there is still using 0.94 and is not planning to upgrade to 0.98 or later soon (which will required downtime), please speak up.
> Otherwise it might be time to think about EOL'ing 0.94.
>
> It's not actually much work to do these releases, especially when they are so small, but I'd like to continue only if they are actually used.
> In any case, I am going to spin 0.94.26 with the current 4 fixes today or tomorrow.
>
> -- Lars
Re: 0.94 going forward
Posted by lars hofhansl <la...@apache.org>.
Yep. That's also why I've doing 0.94 release all this time. 0.92 had a no-downtime path to 0.94. And 0.96 had a no downtime path to 0.98. So both could be EOL'ed with relatively little annoyance.0.94 is different as going to 0.96 or later (including 0.98) is a big change and requires downtime.
If there's a desire I'm happy to continue to do 0.94 releases for a while. Backporting fixes will become more tedious over time, though, as the code lines diverge as time goes on.Maybe a bigger push could be to add a zero-downtime story (somehow) to 0.94, release that in a last release, and then EOL 0.94.
Maybe we can brainstorm how a zero downtime cut over can be done. It's tricky from multiple angles:- replication between 0.94 and 0.98 does not work (there's a gateway process that supposedly does that, but it's not known to be very reliable)- snapshots cannot be exported from 0.94 and 0.98 (please correct me if I'm wrong here)- clients cannot load the 0.94 and 0.98 client at the same into the same JVM (classloader to be specific, so if have OSGi you might be OK)
I'd be happy to hear what design you have in mind.
Thrift should still work between both version in the same way - we'd need to confirm this.Matteo, if you're listening... Do you have an inkling about how hard it would be to export a 0.94 snapshot to 0.98?
-- Lars
From: Jeremy Carroll <ph...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Monday, December 15, 2014 4:49 PM
Subject: Re: 0.94 going forward
Which is why I feel that a lot of customers are still on 0.94. Pretty much
trapped unless you want to take downtime for your site. Any type of
guidance would be helpful. We are currently in the process of designing our
own system to deal with this.
On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <ap...@apache.org> wrote:
>
> Zero downtime upgrade from 0.94 won't be possible. See
> http://hbase.apache.org/book.html#d0e5199
>
>
> On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <ph...@gmail.com>
> wrote:
> >
> > Looking for guidance on how to do a zero downtime upgrade from 0.94 ->
> 0.98
> > (or 1.0 if it launches soon). As soon as we can figure this out, we will
> > migrate over.
> >
> > On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <esteban@cloudera.com
> >
> > wrote:
> > >
> > > Hi Lars,
> > >
> > > Thanks for bringing this for discussion. From my experience I can tell
> > that
> > > 0.94 is very stable but that shouldn't be a blocker to consider to
> > EOL'ing.
> > > Are you considering any specific timeframe for that?
> > >
> > > thanks,
> > > esteban.
> > >
> > > --
> > > Cloudera, Inc.
> > >
> > >
> > > On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com>
> > wrote:
> > > >
> > > > given that CDH4 is hbase 0.94 i dont believe nobody is using it. for
> > our
> > > > clients the majority is on 0.94 (versus 0.96 and up).
> > > >
> > > > so i am going with 1), its very stable!
> > > >
> > > > On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > > > >
> > > > > Over the past few months the rate of the change into 0.94 has
> slowed
> > > > > significantly.
> > > > > 0.94.25 was released on Nov 15th, and since then we had only 4
> > changes.
> > > > >
> > > > > This could mean two things: (1) 0.94 is very stable now or (2)
> nobody
> > > is
> > > > > using it (at least nobody is contributing to it anymore).
> > > > >
> > > > >
> > > > > If anybody out there is still using 0.94 and is not planning to
> > upgrade
> > > > to
> > > > > 0.98 or later soon (which will required downtime), please speak up.
> > > > > Otherwise it might be time to think about EOL'ing 0.94.
> > > > >
> > > > > It's not actually much work to do these releases, especially when
> > they
> > > > are
> > > > > so small, but I'd like to continue only if they are actually used.
> > > > > In any case, I am going to spin 0.94.26 with the current 4 fixes
> > today
> > > or
> > > > > tomorrow.
> > > > >
> > > > > -- Lars
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: 0.94 going forward
Posted by Nick Dimiduk <nd...@gmail.com>.
On Tue, Dec 16, 2014 at 11:53 AM, Ted Yu <yu...@gmail.com> wrote:
> However, the application needs to compiled with 0.98 jars because the RPC
> has changed.
>
It should be possible to build an application that's using both client
library versions using something like JarJar [0] to munge class names. I
believe Jeffrey has done exactly this in the replication bridge [1].
[0]: https://code.google.com/p/jarjar/
[1]: https://github.com/hortonworks/HBaseReplicationBridgeServer
On Tue, Dec 16, 2014 at 11:42 AM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
> >
> > +1 Andrew. Is not a simple task and is error prone and can cause dataloss
> > if not performed correctly, also we don't have tooling to fix broken
> > snapshots if moved manually.
> >
> > BTW 0.98 should migrate an old snapshot dir to the new post-namespaces
> > directory hierarchy after starting HBase from a 0.94 layout. If the goal
> is
> > to minimize downtime probably this should be a better approach: bootstrap
> > the destination cluster with 0.94 with snapshots and replication enabled,
> > then use the ExportSnapshot tool to copy the snapshots, import the
> > snapshots and use replication on the remote cluster until the delta is
> > minimal. Then stop the destination cluster and upgrade to 0.98 (that
> should
> > take care of migrating everything without user intervention). Once the
> > destination cluster is migrated to 0.98 then use the replication bridge
> > tool to catch up with the new edits and reduce the delta between both
> > clusters. Then just fail over the applications to the 0.98 cluster. The
> > repeat the upgrade process in the 0.94 cluster.
> >
> > my 2 cents,
> > esteban.
> >
> >
> >
> >
> >
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Tue, Dec 16, 2014 at 9:37 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> > >
> > > I disagree. Before adding something like that to the ref guide, we
> should
> > > actually agree to support it as a migration strategy. We're not there
> > yet.
> > > And although it's a heroic process, we can take steps to make it less
> > > kludgy if so.
> > >
> > > On Tue, Dec 16, 2014 at 9:27 AM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > Good summary, Brian.
> > > >
> > > > This should be added to ref guide.
> > > >
> > > > Cheers
> > > >
> > > > On Tue, Dec 16, 2014 at 4:17 AM, Brian Jeltema <
> > > > brian.jeltema@digitalenvoy.net> wrote:
> > > > >
> > > > > I have been able to export snapshots from 0.94 to 0.98. I’ve pasted
> > the
> > > > > instructions that I developed
> > > > >
> > > > > and published on our internal wiki. I also had to significantly
> > > increase
> > > > > retry count parameters due to
> > > > >
> > > > > a high number of timeout failures during the export.
> > > > >
> > > > >
> > > > > Cross-cluster transfers
> > > > >
> > > > > To export a snapshot to a different cluster:
> > > > >
> > > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > > snappy
> > > > > -copy-to proto://remhost/apps/hbase/data -mappers nummaps
> > > > >
> > > > > where snappy is the local snapshot to export, remhost is the host
> > name
> > > > the
> > > > > default filesystem in the remote cluster which is the target of the
> > > > export
> > > > > (i.e. the value of fs.defaultFS), nummaps is the number of mappers
> to
> > > run
> > > > > to perform the export, and proto is the protocol to use, either
> hftp,
> > > > hdfs
> > > > > or webhdfs. Use hdfs if the clusters are compatible. Run this as
> the
> > > > hbase
> > > > > user. If you see exceptions being thrown during the transfer
> related
> > to
> > > > > lease expirations, reduce the number of mappers or try using the
> > > > -bandwidth
> > > > > option. You may also see many "File does not exist" warnings in the
> > log
> > > > > output. These can be displayed continuously for several minutes
> for a
> > > > large
> > > > > table, but appear to be noise and can be ignored, so be patient.
> > > However,
> > > > > it is also very common for this command to fail due to a variety of
> > > file
> > > > > ownership conflicts, so you may need to fiddle to get everything
> > > right. A
> > > > > failure of this command often leaves garbage on the target system
> > that
> > > > must
> > > > > be deleted; if that is the case, the command will fail with info on
> > > what
> > > > > needs to be cleaned up.
> > > > >
> > > > > If exporting from an HBase 0.94 cluster to an HBase 0.98 cluster,
> you
> > > > will
> > > > > need to use the webhdfs protocol (or possibly hftp, though I
> couldn’t
> > > get
> > > > > that to work). You also need to manually move some files around
> > because
> > > > > snapshot layouts have changed. Based on the example above, on the
> > 0.98
> > > > > cluster do the following:
> > > > >
> > > > > check whether any imports already exist for the table:
> > > > >
> > > > > hadoop fs -ls /apps/hbase/data/archive/data/default
> > > > >
> > > > > and check whether snappyTable is listed, where snappyTable is
> the
> > > > > source of the snapshot (e.g. hosts). If the source table is listed,
> > > then
> > > > > merge the new ssnapshot data into the existing snapshot data:
> > > > >
> > > > > hadoop fs -mv /apps/hbase/data/.archive/snappyTable/*
> > > > > /apps/hbase/data/archive/data/default/snappyTable
> > > > >
> > > > > hadoop fs -rm r /apps/hbase/data/.archive/snappyTable
> > > > >
> > > > > otherwise, create and populate the snapshot data directory:
> > > > >
> > > > > hadoop fs -mv /apps/hbase/data/.archive/snappyTable
> > > > > /apps/hbase/data/archive/data/default (snappyTable is the source of
> > > > snappy)
> > > > >
> > > > > in either case, update the snapshot metadata files as follows:
> > > > >
> > > > > hadoop fs -mkdir
> > > /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
> > > > >
> > > > > hadoop fs -mv
> > > > > /apps/hbase/data/.hbase-snapshot/snappy/.tableinfo.0000000001
> > > > > /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
> > > > >
> > > > > at this point, you should be able to do a restore_snapshot from
> the
> > > > HBase
> > > > > shell.
> > > > >
> > > > >
> > > > > On Dec 15, 2014, at 8:52 PM, lars hofhansl <la...@apache.org>
> wrote:
> > > > >
> > > > > > Nope :(Replication uses RPC and that was changed to protobufs.
> > AFAIK
> > > > > snapshots can also not be exported from 0.94 and 0.98. We have a
> > really
> > > > > shitty story here. From: Sean Busbey <bu...@cloudera.com>
> > > > > > To: user <us...@hbase.apache.org>
> > > > > > Sent: Monday, December 15, 2014 5:04 PM
> > > > > > Subject: Re: 0.94 going forward
> > > > > >
> > > > > > Does replication and snapshot export work from 0.94.6+ to a 0.96
> or
> > > > 0.98
> > > > > > cluster?
> > > > > >
> > > > > > Presuming it does, shouldn't a site be able to use a multiple
> > cluster
> > > > set
> > > > > > up to do a cut over of a client application?
> > > > > >
> > > > > > That doesn't help with needing downtime for to do the eventual
> > > upgrade,
> > > > > but
> > > > > > it mitigates the impact on the downstream app.
> > > > > >
> > > > > > --
> > > > > > Sean
> > > > > >
> > > > > >
> > > > > > On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com>
> > > wrote:
> > > > > >
> > > > > >> Which is why I feel that a lot of customers are still on 0.94.
> > > Pretty
> > > > > much
> > > > > >> trapped unless you want to take downtime for your site. Any type
> > of
> > > > > >> guidance would be helpful. We are currently in the process of
> > > > designing
> > > > > our
> > > > > >> own system to deal with this.
> > > > > >>
> > > > > >> On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <
> > > apurtell@apache.org>
> > > > > >> wrote:
> > > > > >>>
> > > > > >>> Zero downtime upgrade from 0.94 won't be possible. See
> > > > > >>> http://hbase.apache.org/book.html#d0e5199
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <
> > > phobos182@gmail.com
> > > > >
> > > > > >>> wrote:
> > > > > >>>>
> > > > > >>>> Looking for guidance on how to do a zero downtime upgrade from
> > > 0.94
> > > > ->
> > > > > >>> 0.98
> > > > > >>>> (or 1.0 if it launches soon). As soon as we can figure this
> out,
> > > we
> > > > > >> will
> > > > > >>>> migrate over.
> > > > > >>>>
> > > > > >>>> On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
> > > > > >> esteban@cloudera.com
> > > > > >>>>
> > > > > >>>> wrote:
> > > > > >>>>>
> > > > > >>>>> Hi Lars,
> > > > > >>>>>
> > > > > >>>>> Thanks for bringing this for discussion. From my experience I
> > can
> > > > > >> tell
> > > > > >>>> that
> > > > > >>>>> 0.94 is very stable but that shouldn't be a blocker to
> consider
> > > to
> > > > > >>>> EOL'ing.
> > > > > >>>>> Are you considering any specific timeframe for that?
> > > > > >>>>>
> > > > > >>>>> thanks,
> > > > > >>>>> esteban.
> > > > > >>>>>
> > > > > >>>>> --
> > > > > >>>>> Cloudera, Inc.
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <
> > > koert@tresata.com
> > > > >
> > > > > >>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>> given that CDH4 is hbase 0.94 i dont believe nobody is using
> > it.
> > > > > >> for
> > > > > >>>> our
> > > > > >>>>>> clients the majority is on 0.94 (versus 0.96 and up).
> > > > > >>>>>>
> > > > > >>>>>> so i am going with 1), its very stable!
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <
> > > larsh@apache.org>
> > > > > >>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>> Over the past few months the rate of the change into 0.94
> has
> > > > > >>> slowed
> > > > > >>>>>>> significantly.
> > > > > >>>>>>> 0.94.25 was released on Nov 15th, and since then we had
> only
> > 4
> > > > > >>>> changes.
> > > > > >>>>>>>
> > > > > >>>>>>> This could mean two things: (1) 0.94 is very stable now or
> > (2)
> > > > > >>> nobody
> > > > > >>>>> is
> > > > > >>>>>>> using it (at least nobody is contributing to it anymore).
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> If anybody out there is still using 0.94 and is not
> planning
> > to
> > > > > >>>> upgrade
> > > > > >>>>>> to
> > > > > >>>>>>> 0.98 or later soon (which will required downtime), please
> > speak
> > > > > >> up.
> > > > > >>>>>>> Otherwise it might be time to think about EOL'ing 0.94.
> > > > > >>>>>>>
> > > > > >>>>>>> It's not actually much work to do these releases,
> especially
> > > when
> > > > > >>>> they
> > > > > >>>>>> are
> > > > > >>>>>>> so small, but I'd like to continue only if they are
> actually
> > > > > >> used.
> > > > > >>>>>>> In any case, I am going to spin 0.94.26 with the current 4
> > > fixes
> > > > > >>>> today
> > > > > >>>>> or
> > > > > >>>>>>> tomorrow.
> > > > > >>>>>>>
> > > > > >>>>>>> -- Lars
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> Best regards,
> > > > > >>>
> > > > > >>> - Andy
> > > > > >>>
> > > > > >>> Problems worthy of attack prove their worth by hitting back. -
> > Piet
> > > > > Hein
> > > > > >>> (via Tom White)
> > > > > >>>
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
Re: 0.94 going forward
Posted by Ted Yu <yu...@gmail.com>.
bq. Then just fail over the applications to the 0.98 cluster
However, the application needs to compiled with 0.98 jars because the RPC
has changed.
Cheers
On Tue, Dec 16, 2014 at 11:42 AM, Esteban Gutierrez <es...@cloudera.com>
wrote:
>
> +1 Andrew. Is not a simple task and is error prone and can cause dataloss
> if not performed correctly, also we don't have tooling to fix broken
> snapshots if moved manually.
>
> BTW 0.98 should migrate an old snapshot dir to the new post-namespaces
> directory hierarchy after starting HBase from a 0.94 layout. If the goal is
> to minimize downtime probably this should be a better approach: bootstrap
> the destination cluster with 0.94 with snapshots and replication enabled,
> then use the ExportSnapshot tool to copy the snapshots, import the
> snapshots and use replication on the remote cluster until the delta is
> minimal. Then stop the destination cluster and upgrade to 0.98 (that should
> take care of migrating everything without user intervention). Once the
> destination cluster is migrated to 0.98 then use the replication bridge
> tool to catch up with the new edits and reduce the delta between both
> clusters. Then just fail over the applications to the 0.98 cluster. The
> repeat the upgrade process in the 0.94 cluster.
>
> my 2 cents,
> esteban.
>
>
>
>
>
>
> --
> Cloudera, Inc.
>
>
> On Tue, Dec 16, 2014 at 9:37 AM, Andrew Purtell <ap...@apache.org>
> wrote:
> >
> > I disagree. Before adding something like that to the ref guide, we should
> > actually agree to support it as a migration strategy. We're not there
> yet.
> > And although it's a heroic process, we can take steps to make it less
> > kludgy if so.
> >
> > On Tue, Dec 16, 2014 at 9:27 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > Good summary, Brian.
> > >
> > > This should be added to ref guide.
> > >
> > > Cheers
> > >
> > > On Tue, Dec 16, 2014 at 4:17 AM, Brian Jeltema <
> > > brian.jeltema@digitalenvoy.net> wrote:
> > > >
> > > > I have been able to export snapshots from 0.94 to 0.98. I’ve pasted
> the
> > > > instructions that I developed
> > > >
> > > > and published on our internal wiki. I also had to significantly
> > increase
> > > > retry count parameters due to
> > > >
> > > > a high number of timeout failures during the export.
> > > >
> > > >
> > > > Cross-cluster transfers
> > > >
> > > > To export a snapshot to a different cluster:
> > > >
> > > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> > snappy
> > > > -copy-to proto://remhost/apps/hbase/data -mappers nummaps
> > > >
> > > > where snappy is the local snapshot to export, remhost is the host
> name
> > > the
> > > > default filesystem in the remote cluster which is the target of the
> > > export
> > > > (i.e. the value of fs.defaultFS), nummaps is the number of mappers to
> > run
> > > > to perform the export, and proto is the protocol to use, either hftp,
> > > hdfs
> > > > or webhdfs. Use hdfs if the clusters are compatible. Run this as the
> > > hbase
> > > > user. If you see exceptions being thrown during the transfer related
> to
> > > > lease expirations, reduce the number of mappers or try using the
> > > -bandwidth
> > > > option. You may also see many "File does not exist" warnings in the
> log
> > > > output. These can be displayed continuously for several minutes for a
> > > large
> > > > table, but appear to be noise and can be ignored, so be patient.
> > However,
> > > > it is also very common for this command to fail due to a variety of
> > file
> > > > ownership conflicts, so you may need to fiddle to get everything
> > right. A
> > > > failure of this command often leaves garbage on the target system
> that
> > > must
> > > > be deleted; if that is the case, the command will fail with info on
> > what
> > > > needs to be cleaned up.
> > > >
> > > > If exporting from an HBase 0.94 cluster to an HBase 0.98 cluster, you
> > > will
> > > > need to use the webhdfs protocol (or possibly hftp, though I couldn’t
> > get
> > > > that to work). You also need to manually move some files around
> because
> > > > snapshot layouts have changed. Based on the example above, on the
> 0.98
> > > > cluster do the following:
> > > >
> > > > check whether any imports already exist for the table:
> > > >
> > > > hadoop fs -ls /apps/hbase/data/archive/data/default
> > > >
> > > > and check whether snappyTable is listed, where snappyTable is the
> > > > source of the snapshot (e.g. hosts). If the source table is listed,
> > then
> > > > merge the new ssnapshot data into the existing snapshot data:
> > > >
> > > > hadoop fs -mv /apps/hbase/data/.archive/snappyTable/*
> > > > /apps/hbase/data/archive/data/default/snappyTable
> > > >
> > > > hadoop fs -rm r /apps/hbase/data/.archive/snappyTable
> > > >
> > > > otherwise, create and populate the snapshot data directory:
> > > >
> > > > hadoop fs -mv /apps/hbase/data/.archive/snappyTable
> > > > /apps/hbase/data/archive/data/default (snappyTable is the source of
> > > snappy)
> > > >
> > > > in either case, update the snapshot metadata files as follows:
> > > >
> > > > hadoop fs -mkdir
> > /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
> > > >
> > > > hadoop fs -mv
> > > > /apps/hbase/data/.hbase-snapshot/snappy/.tableinfo.0000000001
> > > > /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
> > > >
> > > > at this point, you should be able to do a restore_snapshot from the
> > > HBase
> > > > shell.
> > > >
> > > >
> > > > On Dec 15, 2014, at 8:52 PM, lars hofhansl <la...@apache.org> wrote:
> > > >
> > > > > Nope :(Replication uses RPC and that was changed to protobufs.
> AFAIK
> > > > snapshots can also not be exported from 0.94 and 0.98. We have a
> really
> > > > shitty story here. From: Sean Busbey <bu...@cloudera.com>
> > > > > To: user <us...@hbase.apache.org>
> > > > > Sent: Monday, December 15, 2014 5:04 PM
> > > > > Subject: Re: 0.94 going forward
> > > > >
> > > > > Does replication and snapshot export work from 0.94.6+ to a 0.96 or
> > > 0.98
> > > > > cluster?
> > > > >
> > > > > Presuming it does, shouldn't a site be able to use a multiple
> cluster
> > > set
> > > > > up to do a cut over of a client application?
> > > > >
> > > > > That doesn't help with needing downtime for to do the eventual
> > upgrade,
> > > > but
> > > > > it mitigates the impact on the downstream app.
> > > > >
> > > > > --
> > > > > Sean
> > > > >
> > > > >
> > > > > On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com>
> > wrote:
> > > > >
> > > > >> Which is why I feel that a lot of customers are still on 0.94.
> > Pretty
> > > > much
> > > > >> trapped unless you want to take downtime for your site. Any type
> of
> > > > >> guidance would be helpful. We are currently in the process of
> > > designing
> > > > our
> > > > >> own system to deal with this.
> > > > >>
> > > > >> On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <
> > apurtell@apache.org>
> > > > >> wrote:
> > > > >>>
> > > > >>> Zero downtime upgrade from 0.94 won't be possible. See
> > > > >>> http://hbase.apache.org/book.html#d0e5199
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <
> > phobos182@gmail.com
> > > >
> > > > >>> wrote:
> > > > >>>>
> > > > >>>> Looking for guidance on how to do a zero downtime upgrade from
> > 0.94
> > > ->
> > > > >>> 0.98
> > > > >>>> (or 1.0 if it launches soon). As soon as we can figure this out,
> > we
> > > > >> will
> > > > >>>> migrate over.
> > > > >>>>
> > > > >>>> On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
> > > > >> esteban@cloudera.com
> > > > >>>>
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>> Hi Lars,
> > > > >>>>>
> > > > >>>>> Thanks for bringing this for discussion. From my experience I
> can
> > > > >> tell
> > > > >>>> that
> > > > >>>>> 0.94 is very stable but that shouldn't be a blocker to consider
> > to
> > > > >>>> EOL'ing.
> > > > >>>>> Are you considering any specific timeframe for that?
> > > > >>>>>
> > > > >>>>> thanks,
> > > > >>>>> esteban.
> > > > >>>>>
> > > > >>>>> --
> > > > >>>>> Cloudera, Inc.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <
> > koert@tresata.com
> > > >
> > > > >>>> wrote:
> > > > >>>>>>
> > > > >>>>>> given that CDH4 is hbase 0.94 i dont believe nobody is using
> it.
> > > > >> for
> > > > >>>> our
> > > > >>>>>> clients the majority is on 0.94 (versus 0.96 and up).
> > > > >>>>>>
> > > > >>>>>> so i am going with 1), its very stable!
> > > > >>>>>>
> > > > >>>>>> On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <
> > larsh@apache.org>
> > > > >>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>> Over the past few months the rate of the change into 0.94 has
> > > > >>> slowed
> > > > >>>>>>> significantly.
> > > > >>>>>>> 0.94.25 was released on Nov 15th, and since then we had only
> 4
> > > > >>>> changes.
> > > > >>>>>>>
> > > > >>>>>>> This could mean two things: (1) 0.94 is very stable now or
> (2)
> > > > >>> nobody
> > > > >>>>> is
> > > > >>>>>>> using it (at least nobody is contributing to it anymore).
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> If anybody out there is still using 0.94 and is not planning
> to
> > > > >>>> upgrade
> > > > >>>>>> to
> > > > >>>>>>> 0.98 or later soon (which will required downtime), please
> speak
> > > > >> up.
> > > > >>>>>>> Otherwise it might be time to think about EOL'ing 0.94.
> > > > >>>>>>>
> > > > >>>>>>> It's not actually much work to do these releases, especially
> > when
> > > > >>>> they
> > > > >>>>>> are
> > > > >>>>>>> so small, but I'd like to continue only if they are actually
> > > > >> used.
> > > > >>>>>>> In any case, I am going to spin 0.94.26 with the current 4
> > fixes
> > > > >>>> today
> > > > >>>>> or
> > > > >>>>>>> tomorrow.
> > > > >>>>>>>
> > > > >>>>>>> -- Lars
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Best regards,
> > > > >>>
> > > > >>> - Andy
> > > > >>>
> > > > >>> Problems worthy of attack prove their worth by hitting back. -
> Piet
> > > > Hein
> > > > >>> (via Tom White)
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: 0.94 going forward
Posted by Esteban Gutierrez <es...@cloudera.com>.
+1 Andrew. Is not a simple task and is error prone and can cause dataloss
if not performed correctly, also we don't have tooling to fix broken
snapshots if moved manually.
BTW 0.98 should migrate an old snapshot dir to the new post-namespaces
directory hierarchy after starting HBase from a 0.94 layout. If the goal is
to minimize downtime probably this should be a better approach: bootstrap
the destination cluster with 0.94 with snapshots and replication enabled,
then use the ExportSnapshot tool to copy the snapshots, import the
snapshots and use replication on the remote cluster until the delta is
minimal. Then stop the destination cluster and upgrade to 0.98 (that should
take care of migrating everything without user intervention). Once the
destination cluster is migrated to 0.98 then use the replication bridge
tool to catch up with the new edits and reduce the delta between both
clusters. Then just fail over the applications to the 0.98 cluster. The
repeat the upgrade process in the 0.94 cluster.
my 2 cents,
esteban.
--
Cloudera, Inc.
On Tue, Dec 16, 2014 at 9:37 AM, Andrew Purtell <ap...@apache.org> wrote:
>
> I disagree. Before adding something like that to the ref guide, we should
> actually agree to support it as a migration strategy. We're not there yet.
> And although it's a heroic process, we can take steps to make it less
> kludgy if so.
>
> On Tue, Dec 16, 2014 at 9:27 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > Good summary, Brian.
> >
> > This should be added to ref guide.
> >
> > Cheers
> >
> > On Tue, Dec 16, 2014 at 4:17 AM, Brian Jeltema <
> > brian.jeltema@digitalenvoy.net> wrote:
> > >
> > > I have been able to export snapshots from 0.94 to 0.98. I’ve pasted the
> > > instructions that I developed
> > >
> > > and published on our internal wiki. I also had to significantly
> increase
> > > retry count parameters due to
> > >
> > > a high number of timeout failures during the export.
> > >
> > >
> > > Cross-cluster transfers
> > >
> > > To export a snapshot to a different cluster:
> > >
> > > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
> snappy
> > > -copy-to proto://remhost/apps/hbase/data -mappers nummaps
> > >
> > > where snappy is the local snapshot to export, remhost is the host name
> > the
> > > default filesystem in the remote cluster which is the target of the
> > export
> > > (i.e. the value of fs.defaultFS), nummaps is the number of mappers to
> run
> > > to perform the export, and proto is the protocol to use, either hftp,
> > hdfs
> > > or webhdfs. Use hdfs if the clusters are compatible. Run this as the
> > hbase
> > > user. If you see exceptions being thrown during the transfer related to
> > > lease expirations, reduce the number of mappers or try using the
> > -bandwidth
> > > option. You may also see many "File does not exist" warnings in the log
> > > output. These can be displayed continuously for several minutes for a
> > large
> > > table, but appear to be noise and can be ignored, so be patient.
> However,
> > > it is also very common for this command to fail due to a variety of
> file
> > > ownership conflicts, so you may need to fiddle to get everything
> right. A
> > > failure of this command often leaves garbage on the target system that
> > must
> > > be deleted; if that is the case, the command will fail with info on
> what
> > > needs to be cleaned up.
> > >
> > > If exporting from an HBase 0.94 cluster to an HBase 0.98 cluster, you
> > will
> > > need to use the webhdfs protocol (or possibly hftp, though I couldn’t
> get
> > > that to work). You also need to manually move some files around because
> > > snapshot layouts have changed. Based on the example above, on the 0.98
> > > cluster do the following:
> > >
> > > check whether any imports already exist for the table:
> > >
> > > hadoop fs -ls /apps/hbase/data/archive/data/default
> > >
> > > and check whether snappyTable is listed, where snappyTable is the
> > > source of the snapshot (e.g. hosts). If the source table is listed,
> then
> > > merge the new ssnapshot data into the existing snapshot data:
> > >
> > > hadoop fs -mv /apps/hbase/data/.archive/snappyTable/*
> > > /apps/hbase/data/archive/data/default/snappyTable
> > >
> > > hadoop fs -rm r /apps/hbase/data/.archive/snappyTable
> > >
> > > otherwise, create and populate the snapshot data directory:
> > >
> > > hadoop fs -mv /apps/hbase/data/.archive/snappyTable
> > > /apps/hbase/data/archive/data/default (snappyTable is the source of
> > snappy)
> > >
> > > in either case, update the snapshot metadata files as follows:
> > >
> > > hadoop fs -mkdir
> /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
> > >
> > > hadoop fs -mv
> > > /apps/hbase/data/.hbase-snapshot/snappy/.tableinfo.0000000001
> > > /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
> > >
> > > at this point, you should be able to do a restore_snapshot from the
> > HBase
> > > shell.
> > >
> > >
> > > On Dec 15, 2014, at 8:52 PM, lars hofhansl <la...@apache.org> wrote:
> > >
> > > > Nope :(Replication uses RPC and that was changed to protobufs. AFAIK
> > > snapshots can also not be exported from 0.94 and 0.98. We have a really
> > > shitty story here. From: Sean Busbey <bu...@cloudera.com>
> > > > To: user <us...@hbase.apache.org>
> > > > Sent: Monday, December 15, 2014 5:04 PM
> > > > Subject: Re: 0.94 going forward
> > > >
> > > > Does replication and snapshot export work from 0.94.6+ to a 0.96 or
> > 0.98
> > > > cluster?
> > > >
> > > > Presuming it does, shouldn't a site be able to use a multiple cluster
> > set
> > > > up to do a cut over of a client application?
> > > >
> > > > That doesn't help with needing downtime for to do the eventual
> upgrade,
> > > but
> > > > it mitigates the impact on the downstream app.
> > > >
> > > > --
> > > > Sean
> > > >
> > > >
> > > > On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com>
> wrote:
> > > >
> > > >> Which is why I feel that a lot of customers are still on 0.94.
> Pretty
> > > much
> > > >> trapped unless you want to take downtime for your site. Any type of
> > > >> guidance would be helpful. We are currently in the process of
> > designing
> > > our
> > > >> own system to deal with this.
> > > >>
> > > >> On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <
> apurtell@apache.org>
> > > >> wrote:
> > > >>>
> > > >>> Zero downtime upgrade from 0.94 won't be possible. See
> > > >>> http://hbase.apache.org/book.html#d0e5199
> > > >>>
> > > >>>
> > > >>> On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <
> phobos182@gmail.com
> > >
> > > >>> wrote:
> > > >>>>
> > > >>>> Looking for guidance on how to do a zero downtime upgrade from
> 0.94
> > ->
> > > >>> 0.98
> > > >>>> (or 1.0 if it launches soon). As soon as we can figure this out,
> we
> > > >> will
> > > >>>> migrate over.
> > > >>>>
> > > >>>> On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
> > > >> esteban@cloudera.com
> > > >>>>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>> Hi Lars,
> > > >>>>>
> > > >>>>> Thanks for bringing this for discussion. From my experience I can
> > > >> tell
> > > >>>> that
> > > >>>>> 0.94 is very stable but that shouldn't be a blocker to consider
> to
> > > >>>> EOL'ing.
> > > >>>>> Are you considering any specific timeframe for that?
> > > >>>>>
> > > >>>>> thanks,
> > > >>>>> esteban.
> > > >>>>>
> > > >>>>> --
> > > >>>>> Cloudera, Inc.
> > > >>>>>
> > > >>>>>
> > > >>>>> On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <
> koert@tresata.com
> > >
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>> given that CDH4 is hbase 0.94 i dont believe nobody is using it.
> > > >> for
> > > >>>> our
> > > >>>>>> clients the majority is on 0.94 (versus 0.96 and up).
> > > >>>>>>
> > > >>>>>> so i am going with 1), its very stable!
> > > >>>>>>
> > > >>>>>> On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <
> larsh@apache.org>
> > > >>>> wrote:
> > > >>>>>>>
> > > >>>>>>> Over the past few months the rate of the change into 0.94 has
> > > >>> slowed
> > > >>>>>>> significantly.
> > > >>>>>>> 0.94.25 was released on Nov 15th, and since then we had only 4
> > > >>>> changes.
> > > >>>>>>>
> > > >>>>>>> This could mean two things: (1) 0.94 is very stable now or (2)
> > > >>> nobody
> > > >>>>> is
> > > >>>>>>> using it (at least nobody is contributing to it anymore).
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> If anybody out there is still using 0.94 and is not planning to
> > > >>>> upgrade
> > > >>>>>> to
> > > >>>>>>> 0.98 or later soon (which will required downtime), please speak
> > > >> up.
> > > >>>>>>> Otherwise it might be time to think about EOL'ing 0.94.
> > > >>>>>>>
> > > >>>>>>> It's not actually much work to do these releases, especially
> when
> > > >>>> they
> > > >>>>>> are
> > > >>>>>>> so small, but I'd like to continue only if they are actually
> > > >> used.
> > > >>>>>>> In any case, I am going to spin 0.94.26 with the current 4
> fixes
> > > >>>> today
> > > >>>>> or
> > > >>>>>>> tomorrow.
> > > >>>>>>>
> > > >>>>>>> -- Lars
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Best regards,
> > > >>>
> > > >>> - Andy
> > > >>>
> > > >>> Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein
> > > >>> (via Tom White)
> > > >>>
> > > >>
> > > >
> > > >
> > >
> > >
> > >
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: 0.94 going forward
Posted by Andrew Purtell <ap...@apache.org>.
I disagree. Before adding something like that to the ref guide, we should
actually agree to support it as a migration strategy. We're not there yet.
And although it's a heroic process, we can take steps to make it less
kludgy if so.
On Tue, Dec 16, 2014 at 9:27 AM, Ted Yu <yu...@gmail.com> wrote:
>
> Good summary, Brian.
>
> This should be added to ref guide.
>
> Cheers
>
> On Tue, Dec 16, 2014 at 4:17 AM, Brian Jeltema <
> brian.jeltema@digitalenvoy.net> wrote:
> >
> > I have been able to export snapshots from 0.94 to 0.98. I’ve pasted the
> > instructions that I developed
> >
> > and published on our internal wiki. I also had to significantly increase
> > retry count parameters due to
> >
> > a high number of timeout failures during the export.
> >
> >
> > Cross-cluster transfers
> >
> > To export a snapshot to a different cluster:
> >
> > hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snappy
> > -copy-to proto://remhost/apps/hbase/data -mappers nummaps
> >
> > where snappy is the local snapshot to export, remhost is the host name
> the
> > default filesystem in the remote cluster which is the target of the
> export
> > (i.e. the value of fs.defaultFS), nummaps is the number of mappers to run
> > to perform the export, and proto is the protocol to use, either hftp,
> hdfs
> > or webhdfs. Use hdfs if the clusters are compatible. Run this as the
> hbase
> > user. If you see exceptions being thrown during the transfer related to
> > lease expirations, reduce the number of mappers or try using the
> -bandwidth
> > option. You may also see many "File does not exist" warnings in the log
> > output. These can be displayed continuously for several minutes for a
> large
> > table, but appear to be noise and can be ignored, so be patient. However,
> > it is also very common for this command to fail due to a variety of file
> > ownership conflicts, so you may need to fiddle to get everything right. A
> > failure of this command often leaves garbage on the target system that
> must
> > be deleted; if that is the case, the command will fail with info on what
> > needs to be cleaned up.
> >
> > If exporting from an HBase 0.94 cluster to an HBase 0.98 cluster, you
> will
> > need to use the webhdfs protocol (or possibly hftp, though I couldn’t get
> > that to work). You also need to manually move some files around because
> > snapshot layouts have changed. Based on the example above, on the 0.98
> > cluster do the following:
> >
> > check whether any imports already exist for the table:
> >
> > hadoop fs -ls /apps/hbase/data/archive/data/default
> >
> > and check whether snappyTable is listed, where snappyTable is the
> > source of the snapshot (e.g. hosts). If the source table is listed, then
> > merge the new ssnapshot data into the existing snapshot data:
> >
> > hadoop fs -mv /apps/hbase/data/.archive/snappyTable/*
> > /apps/hbase/data/archive/data/default/snappyTable
> >
> > hadoop fs -rm r /apps/hbase/data/.archive/snappyTable
> >
> > otherwise, create and populate the snapshot data directory:
> >
> > hadoop fs -mv /apps/hbase/data/.archive/snappyTable
> > /apps/hbase/data/archive/data/default (snappyTable is the source of
> snappy)
> >
> > in either case, update the snapshot metadata files as follows:
> >
> > hadoop fs -mkdir /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
> >
> > hadoop fs -mv
> > /apps/hbase/data/.hbase-snapshot/snappy/.tableinfo.0000000001
> > /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
> >
> > at this point, you should be able to do a restore_snapshot from the
> HBase
> > shell.
> >
> >
> > On Dec 15, 2014, at 8:52 PM, lars hofhansl <la...@apache.org> wrote:
> >
> > > Nope :(Replication uses RPC and that was changed to protobufs. AFAIK
> > snapshots can also not be exported from 0.94 and 0.98. We have a really
> > shitty story here. From: Sean Busbey <bu...@cloudera.com>
> > > To: user <us...@hbase.apache.org>
> > > Sent: Monday, December 15, 2014 5:04 PM
> > > Subject: Re: 0.94 going forward
> > >
> > > Does replication and snapshot export work from 0.94.6+ to a 0.96 or
> 0.98
> > > cluster?
> > >
> > > Presuming it does, shouldn't a site be able to use a multiple cluster
> set
> > > up to do a cut over of a client application?
> > >
> > > That doesn't help with needing downtime for to do the eventual upgrade,
> > but
> > > it mitigates the impact on the downstream app.
> > >
> > > --
> > > Sean
> > >
> > >
> > > On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com> wrote:
> > >
> > >> Which is why I feel that a lot of customers are still on 0.94. Pretty
> > much
> > >> trapped unless you want to take downtime for your site. Any type of
> > >> guidance would be helpful. We are currently in the process of
> designing
> > our
> > >> own system to deal with this.
> > >>
> > >> On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <ap...@apache.org>
> > >> wrote:
> > >>>
> > >>> Zero downtime upgrade from 0.94 won't be possible. See
> > >>> http://hbase.apache.org/book.html#d0e5199
> > >>>
> > >>>
> > >>> On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <phobos182@gmail.com
> >
> > >>> wrote:
> > >>>>
> > >>>> Looking for guidance on how to do a zero downtime upgrade from 0.94
> ->
> > >>> 0.98
> > >>>> (or 1.0 if it launches soon). As soon as we can figure this out, we
> > >> will
> > >>>> migrate over.
> > >>>>
> > >>>> On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
> > >> esteban@cloudera.com
> > >>>>
> > >>>> wrote:
> > >>>>>
> > >>>>> Hi Lars,
> > >>>>>
> > >>>>> Thanks for bringing this for discussion. From my experience I can
> > >> tell
> > >>>> that
> > >>>>> 0.94 is very stable but that shouldn't be a blocker to consider to
> > >>>> EOL'ing.
> > >>>>> Are you considering any specific timeframe for that?
> > >>>>>
> > >>>>> thanks,
> > >>>>> esteban.
> > >>>>>
> > >>>>> --
> > >>>>> Cloudera, Inc.
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <koert@tresata.com
> >
> > >>>> wrote:
> > >>>>>>
> > >>>>>> given that CDH4 is hbase 0.94 i dont believe nobody is using it.
> > >> for
> > >>>> our
> > >>>>>> clients the majority is on 0.94 (versus 0.96 and up).
> > >>>>>>
> > >>>>>> so i am going with 1), its very stable!
> > >>>>>>
> > >>>>>> On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org>
> > >>>> wrote:
> > >>>>>>>
> > >>>>>>> Over the past few months the rate of the change into 0.94 has
> > >>> slowed
> > >>>>>>> significantly.
> > >>>>>>> 0.94.25 was released on Nov 15th, and since then we had only 4
> > >>>> changes.
> > >>>>>>>
> > >>>>>>> This could mean two things: (1) 0.94 is very stable now or (2)
> > >>> nobody
> > >>>>> is
> > >>>>>>> using it (at least nobody is contributing to it anymore).
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> If anybody out there is still using 0.94 and is not planning to
> > >>>> upgrade
> > >>>>>> to
> > >>>>>>> 0.98 or later soon (which will required downtime), please speak
> > >> up.
> > >>>>>>> Otherwise it might be time to think about EOL'ing 0.94.
> > >>>>>>>
> > >>>>>>> It's not actually much work to do these releases, especially when
> > >>>> they
> > >>>>>> are
> > >>>>>>> so small, but I'd like to continue only if they are actually
> > >> used.
> > >>>>>>> In any case, I am going to spin 0.94.26 with the current 4 fixes
> > >>>> today
> > >>>>> or
> > >>>>>>> tomorrow.
> > >>>>>>>
> > >>>>>>> -- Lars
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>> Best regards,
> > >>>
> > >>> - Andy
> > >>>
> > >>> Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > >>> (via Tom White)
> > >>>
> > >>
> > >
> > >
> >
> >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: 0.94 going forward
Posted by Ted Yu <yu...@gmail.com>.
Good summary, Brian.
This should be added to ref guide.
Cheers
On Tue, Dec 16, 2014 at 4:17 AM, Brian Jeltema <
brian.jeltema@digitalenvoy.net> wrote:
>
> I have been able to export snapshots from 0.94 to 0.98. I’ve pasted the
> instructions that I developed
>
> and published on our internal wiki. I also had to significantly increase
> retry count parameters due to
>
> a high number of timeout failures during the export.
>
>
> Cross-cluster transfers
>
> To export a snapshot to a different cluster:
>
> hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snappy
> -copy-to proto://remhost/apps/hbase/data -mappers nummaps
>
> where snappy is the local snapshot to export, remhost is the host name the
> default filesystem in the remote cluster which is the target of the export
> (i.e. the value of fs.defaultFS), nummaps is the number of mappers to run
> to perform the export, and proto is the protocol to use, either hftp, hdfs
> or webhdfs. Use hdfs if the clusters are compatible. Run this as the hbase
> user. If you see exceptions being thrown during the transfer related to
> lease expirations, reduce the number of mappers or try using the -bandwidth
> option. You may also see many "File does not exist" warnings in the log
> output. These can be displayed continuously for several minutes for a large
> table, but appear to be noise and can be ignored, so be patient. However,
> it is also very common for this command to fail due to a variety of file
> ownership conflicts, so you may need to fiddle to get everything right. A
> failure of this command often leaves garbage on the target system that must
> be deleted; if that is the case, the command will fail with info on what
> needs to be cleaned up.
>
> If exporting from an HBase 0.94 cluster to an HBase 0.98 cluster, you will
> need to use the webhdfs protocol (or possibly hftp, though I couldn’t get
> that to work). You also need to manually move some files around because
> snapshot layouts have changed. Based on the example above, on the 0.98
> cluster do the following:
>
> check whether any imports already exist for the table:
>
> hadoop fs -ls /apps/hbase/data/archive/data/default
>
> and check whether snappyTable is listed, where snappyTable is the
> source of the snapshot (e.g. hosts). If the source table is listed, then
> merge the new ssnapshot data into the existing snapshot data:
>
> hadoop fs -mv /apps/hbase/data/.archive/snappyTable/*
> /apps/hbase/data/archive/data/default/snappyTable
>
> hadoop fs -rm r /apps/hbase/data/.archive/snappyTable
>
> otherwise, create and populate the snapshot data directory:
>
> hadoop fs -mv /apps/hbase/data/.archive/snappyTable
> /apps/hbase/data/archive/data/default (snappyTable is the source of snappy)
>
> in either case, update the snapshot metadata files as follows:
>
> hadoop fs -mkdir /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
>
> hadoop fs -mv
> /apps/hbase/data/.hbase-snapshot/snappy/.tableinfo.0000000001
> /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
>
> at this point, you should be able to do a restore_snapshot from the HBase
> shell.
>
>
> On Dec 15, 2014, at 8:52 PM, lars hofhansl <la...@apache.org> wrote:
>
> > Nope :(Replication uses RPC and that was changed to protobufs. AFAIK
> snapshots can also not be exported from 0.94 and 0.98. We have a really
> shitty story here. From: Sean Busbey <bu...@cloudera.com>
> > To: user <us...@hbase.apache.org>
> > Sent: Monday, December 15, 2014 5:04 PM
> > Subject: Re: 0.94 going forward
> >
> > Does replication and snapshot export work from 0.94.6+ to a 0.96 or 0.98
> > cluster?
> >
> > Presuming it does, shouldn't a site be able to use a multiple cluster set
> > up to do a cut over of a client application?
> >
> > That doesn't help with needing downtime for to do the eventual upgrade,
> but
> > it mitigates the impact on the downstream app.
> >
> > --
> > Sean
> >
> >
> > On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com> wrote:
> >
> >> Which is why I feel that a lot of customers are still on 0.94. Pretty
> much
> >> trapped unless you want to take downtime for your site. Any type of
> >> guidance would be helpful. We are currently in the process of designing
> our
> >> own system to deal with this.
> >>
> >> On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <ap...@apache.org>
> >> wrote:
> >>>
> >>> Zero downtime upgrade from 0.94 won't be possible. See
> >>> http://hbase.apache.org/book.html#d0e5199
> >>>
> >>>
> >>> On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <ph...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Looking for guidance on how to do a zero downtime upgrade from 0.94 ->
> >>> 0.98
> >>>> (or 1.0 if it launches soon). As soon as we can figure this out, we
> >> will
> >>>> migrate over.
> >>>>
> >>>> On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
> >> esteban@cloudera.com
> >>>>
> >>>> wrote:
> >>>>>
> >>>>> Hi Lars,
> >>>>>
> >>>>> Thanks for bringing this for discussion. From my experience I can
> >> tell
> >>>> that
> >>>>> 0.94 is very stable but that shouldn't be a blocker to consider to
> >>>> EOL'ing.
> >>>>> Are you considering any specific timeframe for that?
> >>>>>
> >>>>> thanks,
> >>>>> esteban.
> >>>>>
> >>>>> --
> >>>>> Cloudera, Inc.
> >>>>>
> >>>>>
> >>>>> On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com>
> >>>> wrote:
> >>>>>>
> >>>>>> given that CDH4 is hbase 0.94 i dont believe nobody is using it.
> >> for
> >>>> our
> >>>>>> clients the majority is on 0.94 (versus 0.96 and up).
> >>>>>>
> >>>>>> so i am going with 1), its very stable!
> >>>>>>
> >>>>>> On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org>
> >>>> wrote:
> >>>>>>>
> >>>>>>> Over the past few months the rate of the change into 0.94 has
> >>> slowed
> >>>>>>> significantly.
> >>>>>>> 0.94.25 was released on Nov 15th, and since then we had only 4
> >>>> changes.
> >>>>>>>
> >>>>>>> This could mean two things: (1) 0.94 is very stable now or (2)
> >>> nobody
> >>>>> is
> >>>>>>> using it (at least nobody is contributing to it anymore).
> >>>>>>>
> >>>>>>>
> >>>>>>> If anybody out there is still using 0.94 and is not planning to
> >>>> upgrade
> >>>>>> to
> >>>>>>> 0.98 or later soon (which will required downtime), please speak
> >> up.
> >>>>>>> Otherwise it might be time to think about EOL'ing 0.94.
> >>>>>>>
> >>>>>>> It's not actually much work to do these releases, especially when
> >>>> they
> >>>>>> are
> >>>>>>> so small, but I'd like to continue only if they are actually
> >> used.
> >>>>>>> In any case, I am going to spin 0.94.26 with the current 4 fixes
> >>>> today
> >>>>> or
> >>>>>>> tomorrow.
> >>>>>>>
> >>>>>>> -- Lars
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Best regards,
> >>>
> >>> - Andy
> >>>
> >>> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >>> (via Tom White)
> >>>
> >>
> >
> >
>
>
>
Re: 0.94 going forward
Posted by Brian Jeltema <br...@digitalenvoy.net>.
I have been able to export snapshots from 0.94 to 0.98. I’ve pasted the instructions that I developed
and published on our internal wiki. I also had to significantly increase retry count parameters due to
a high number of timeout failures during the export.
Cross-cluster transfers
To export a snapshot to a different cluster:
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snappy -copy-to proto://remhost/apps/hbase/data -mappers nummaps
where snappy is the local snapshot to export, remhost is the host name the default filesystem in the remote cluster which is the target of the export (i.e. the value of fs.defaultFS), nummaps is the number of mappers to run to perform the export, and proto is the protocol to use, either hftp, hdfs or webhdfs. Use hdfs if the clusters are compatible. Run this as the hbase user. If you see exceptions being thrown during the transfer related to lease expirations, reduce the number of mappers or try using the -bandwidth option. You may also see many "File does not exist" warnings in the log output. These can be displayed continuously for several minutes for a large table, but appear to be noise and can be ignored, so be patient. However, it is also very common for this command to fail due to a variety of file ownership conflicts, so you may need to fiddle to get everything right. A failure of this command often leaves garbage on the target system that must be deleted; if that is the case, the command will fail with info on what needs to be cleaned up.
If exporting from an HBase 0.94 cluster to an HBase 0.98 cluster, you will need to use the webhdfs protocol (or possibly hftp, though I couldn’t get that to work). You also need to manually move some files around because snapshot layouts have changed. Based on the example above, on the 0.98 cluster do the following:
check whether any imports already exist for the table:
hadoop fs -ls /apps/hbase/data/archive/data/default
and check whether snappyTable is listed, where snappyTable is the source of the snapshot (e.g. hosts). If the source table is listed, then merge the new ssnapshot data into the existing snapshot data:
hadoop fs -mv /apps/hbase/data/.archive/snappyTable/* /apps/hbase/data/archive/data/default/snappyTable
hadoop fs -rm r /apps/hbase/data/.archive/snappyTable
otherwise, create and populate the snapshot data directory:
hadoop fs -mv /apps/hbase/data/.archive/snappyTable /apps/hbase/data/archive/data/default (snappyTable is the source of snappy)
in either case, update the snapshot metadata files as follows:
hadoop fs -mkdir /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
hadoop fs -mv /apps/hbase/data/.hbase-snapshot/snappy/.tableinfo.0000000001 /apps/hbase/data/.hbase-snapshot/snappy/.tabledesc
at this point, you should be able to do a restore_snapshot from the HBase shell.
On Dec 15, 2014, at 8:52 PM, lars hofhansl <la...@apache.org> wrote:
> Nope :(Replication uses RPC and that was changed to protobufs. AFAIK snapshots can also not be exported from 0.94 and 0.98. We have a really shitty story here. From: Sean Busbey <bu...@cloudera.com>
> To: user <us...@hbase.apache.org>
> Sent: Monday, December 15, 2014 5:04 PM
> Subject: Re: 0.94 going forward
>
> Does replication and snapshot export work from 0.94.6+ to a 0.96 or 0.98
> cluster?
>
> Presuming it does, shouldn't a site be able to use a multiple cluster set
> up to do a cut over of a client application?
>
> That doesn't help with needing downtime for to do the eventual upgrade, but
> it mitigates the impact on the downstream app.
>
> --
> Sean
>
>
> On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com> wrote:
>
>> Which is why I feel that a lot of customers are still on 0.94. Pretty much
>> trapped unless you want to take downtime for your site. Any type of
>> guidance would be helpful. We are currently in the process of designing our
>> own system to deal with this.
>>
>> On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>>
>>> Zero downtime upgrade from 0.94 won't be possible. See
>>> http://hbase.apache.org/book.html#d0e5199
>>>
>>>
>>> On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <ph...@gmail.com>
>>> wrote:
>>>>
>>>> Looking for guidance on how to do a zero downtime upgrade from 0.94 ->
>>> 0.98
>>>> (or 1.0 if it launches soon). As soon as we can figure this out, we
>> will
>>>> migrate over.
>>>>
>>>> On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
>> esteban@cloudera.com
>>>>
>>>> wrote:
>>>>>
>>>>> Hi Lars,
>>>>>
>>>>> Thanks for bringing this for discussion. From my experience I can
>> tell
>>>> that
>>>>> 0.94 is very stable but that shouldn't be a blocker to consider to
>>>> EOL'ing.
>>>>> Are you considering any specific timeframe for that?
>>>>>
>>>>> thanks,
>>>>> esteban.
>>>>>
>>>>> --
>>>>> Cloudera, Inc.
>>>>>
>>>>>
>>>>> On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com>
>>>> wrote:
>>>>>>
>>>>>> given that CDH4 is hbase 0.94 i dont believe nobody is using it.
>> for
>>>> our
>>>>>> clients the majority is on 0.94 (versus 0.96 and up).
>>>>>>
>>>>>> so i am going with 1), its very stable!
>>>>>>
>>>>>> On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org>
>>>> wrote:
>>>>>>>
>>>>>>> Over the past few months the rate of the change into 0.94 has
>>> slowed
>>>>>>> significantly.
>>>>>>> 0.94.25 was released on Nov 15th, and since then we had only 4
>>>> changes.
>>>>>>>
>>>>>>> This could mean two things: (1) 0.94 is very stable now or (2)
>>> nobody
>>>>> is
>>>>>>> using it (at least nobody is contributing to it anymore).
>>>>>>>
>>>>>>>
>>>>>>> If anybody out there is still using 0.94 and is not planning to
>>>> upgrade
>>>>>> to
>>>>>>> 0.98 or later soon (which will required downtime), please speak
>> up.
>>>>>>> Otherwise it might be time to think about EOL'ing 0.94.
>>>>>>>
>>>>>>> It's not actually much work to do these releases, especially when
>>>> they
>>>>>> are
>>>>>>> so small, but I'd like to continue only if they are actually
>> used.
>>>>>>> In any case, I am going to spin 0.94.26 with the current 4 fixes
>>>> today
>>>>> or
>>>>>>> tomorrow.
>>>>>>>
>>>>>>> -- Lars
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
>>>
>>
>
>
Re: 0.94 going forward
Posted by lars hofhansl <la...@apache.org>.
Nope :(Replication uses RPC and that was changed to protobufs. AFAIK snapshots can also not be exported from 0.94 and 0.98. We have a really shitty story here. From: Sean Busbey <bu...@cloudera.com>
To: user <us...@hbase.apache.org>
Sent: Monday, December 15, 2014 5:04 PM
Subject: Re: 0.94 going forward
Does replication and snapshot export work from 0.94.6+ to a 0.96 or 0.98
cluster?
Presuming it does, shouldn't a site be able to use a multiple cluster set
up to do a cut over of a client application?
That doesn't help with needing downtime for to do the eventual upgrade, but
it mitigates the impact on the downstream app.
--
Sean
On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com> wrote:
> Which is why I feel that a lot of customers are still on 0.94. Pretty much
> trapped unless you want to take downtime for your site. Any type of
> guidance would be helpful. We are currently in the process of designing our
> own system to deal with this.
>
> On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> >
> > Zero downtime upgrade from 0.94 won't be possible. See
> > http://hbase.apache.org/book.html#d0e5199
> >
> >
> > On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <ph...@gmail.com>
> > wrote:
> > >
> > > Looking for guidance on how to do a zero downtime upgrade from 0.94 ->
> > 0.98
> > > (or 1.0 if it launches soon). As soon as we can figure this out, we
> will
> > > migrate over.
> > >
> > > On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
> esteban@cloudera.com
> > >
> > > wrote:
> > > >
> > > > Hi Lars,
> > > >
> > > > Thanks for bringing this for discussion. From my experience I can
> tell
> > > that
> > > > 0.94 is very stable but that shouldn't be a blocker to consider to
> > > EOL'ing.
> > > > Are you considering any specific timeframe for that?
> > > >
> > > > thanks,
> > > > esteban.
> > > >
> > > > --
> > > > Cloudera, Inc.
> > > >
> > > >
> > > > On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com>
> > > wrote:
> > > > >
> > > > > given that CDH4 is hbase 0.94 i dont believe nobody is using it.
> for
> > > our
> > > > > clients the majority is on 0.94 (versus 0.96 and up).
> > > > >
> > > > > so i am going with 1), its very stable!
> > > > >
> > > > > On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org>
> > > wrote:
> > > > > >
> > > > > > Over the past few months the rate of the change into 0.94 has
> > slowed
> > > > > > significantly.
> > > > > > 0.94.25 was released on Nov 15th, and since then we had only 4
> > > changes.
> > > > > >
> > > > > > This could mean two things: (1) 0.94 is very stable now or (2)
> > nobody
> > > > is
> > > > > > using it (at least nobody is contributing to it anymore).
> > > > > >
> > > > > >
> > > > > > If anybody out there is still using 0.94 and is not planning to
> > > upgrade
> > > > > to
> > > > > > 0.98 or later soon (which will required downtime), please speak
> up.
> > > > > > Otherwise it might be time to think about EOL'ing 0.94.
> > > > > >
> > > > > > It's not actually much work to do these releases, especially when
> > > they
> > > > > are
> > > > > > so small, but I'd like to continue only if they are actually
> used.
> > > > > > In any case, I am going to spin 0.94.26 with the current 4 fixes
> > > today
> > > > or
> > > > > > tomorrow.
> > > > > >
> > > > > > -- Lars
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: 0.94 going forward
Posted by Esteban Gutierrez <es...@cloudera.com>.
Replication from 0.94 to +0.96 wont work out of the box unless you use a
bridge: HBASE-9360 Maybe before EOL'ing 0.94 we should ship it as part of
0.94.
cheers,
esteban.
--
Cloudera, Inc.
On Mon, Dec 15, 2014 at 5:23 PM, Bijieshan <bi...@huawei.com> wrote:
>
> Thanks, Lars. We have customers still using 94. It is indeed stable now.
>
> Jieshan.
> ________________________________________
> From: Sean Busbey [busbey@cloudera.com]
> Sent: Tuesday, December 16, 2014 9:04 AM
> To: user
> Subject: Re: 0.94 going forward
>
> Does replication and snapshot export work from 0.94.6+ to a 0.96 or 0.98
> cluster?
>
> Presuming it does, shouldn't a site be able to use a multiple cluster set
> up to do a cut over of a client application?
>
> That doesn't help with needing downtime for to do the eventual upgrade, but
> it mitigates the impact on the downstream app.
>
> --
> Sean
> On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com> wrote:
>
> > Which is why I feel that a lot of customers are still on 0.94. Pretty
> much
> > trapped unless you want to take downtime for your site. Any type of
> > guidance would be helpful. We are currently in the process of designing
> our
> > own system to deal with this.
> >
> > On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> > >
> > > Zero downtime upgrade from 0.94 won't be possible. See
> > > http://hbase.apache.org/book.html#d0e5199
> > >
> > >
> > > On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <ph...@gmail.com>
> > > wrote:
> > > >
> > > > Looking for guidance on how to do a zero downtime upgrade from 0.94
> ->
> > > 0.98
> > > > (or 1.0 if it launches soon). As soon as we can figure this out, we
> > will
> > > > migrate over.
> > > >
> > > > On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
> > esteban@cloudera.com
> > > >
> > > > wrote:
> > > > >
> > > > > Hi Lars,
> > > > >
> > > > > Thanks for bringing this for discussion. From my experience I can
> > tell
> > > > that
> > > > > 0.94 is very stable but that shouldn't be a blocker to consider to
> > > > EOL'ing.
> > > > > Are you considering any specific timeframe for that?
> > > > >
> > > > > thanks,
> > > > > esteban.
> > > > >
> > > > > --
> > > > > Cloudera, Inc.
> > > > >
> > > > >
> > > > > On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <koert@tresata.com
> >
> > > > wrote:
> > > > > >
> > > > > > given that CDH4 is hbase 0.94 i dont believe nobody is using it.
> > for
> > > > our
> > > > > > clients the majority is on 0.94 (versus 0.96 and up).
> > > > > >
> > > > > > so i am going with 1), its very stable!
> > > > > >
> > > > > > On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <larsh@apache.org
> >
> > > > wrote:
> > > > > > >
> > > > > > > Over the past few months the rate of the change into 0.94 has
> > > slowed
> > > > > > > significantly.
> > > > > > > 0.94.25 was released on Nov 15th, and since then we had only 4
> > > > changes.
> > > > > > >
> > > > > > > This could mean two things: (1) 0.94 is very stable now or (2)
> > > nobody
> > > > > is
> > > > > > > using it (at least nobody is contributing to it anymore).
> > > > > > >
> > > > > > >
> > > > > > > If anybody out there is still using 0.94 and is not planning to
> > > > upgrade
> > > > > > to
> > > > > > > 0.98 or later soon (which will required downtime), please speak
> > up.
> > > > > > > Otherwise it might be time to think about EOL'ing 0.94.
> > > > > > >
> > > > > > > It's not actually much work to do these releases, especially
> when
> > > > they
> > > > > > are
> > > > > > > so small, but I'd like to continue only if they are actually
> > used.
> > > > > > > In any case, I am going to spin 0.94.26 with the current 4
> fixes
> > > > today
> > > > > or
> > > > > > > tomorrow.
> > > > > > >
> > > > > > > -- Lars
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
RE: 0.94 going forward
Posted by Bijieshan <bi...@huawei.com>.
Thanks, Lars. We have customers still using 94. It is indeed stable now.
Jieshan.
________________________________________
From: Sean Busbey [busbey@cloudera.com]
Sent: Tuesday, December 16, 2014 9:04 AM
To: user
Subject: Re: 0.94 going forward
Does replication and snapshot export work from 0.94.6+ to a 0.96 or 0.98
cluster?
Presuming it does, shouldn't a site be able to use a multiple cluster set
up to do a cut over of a client application?
That doesn't help with needing downtime for to do the eventual upgrade, but
it mitigates the impact on the downstream app.
--
Sean
On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com> wrote:
> Which is why I feel that a lot of customers are still on 0.94. Pretty much
> trapped unless you want to take downtime for your site. Any type of
> guidance would be helpful. We are currently in the process of designing our
> own system to deal with this.
>
> On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> >
> > Zero downtime upgrade from 0.94 won't be possible. See
> > http://hbase.apache.org/book.html#d0e5199
> >
> >
> > On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <ph...@gmail.com>
> > wrote:
> > >
> > > Looking for guidance on how to do a zero downtime upgrade from 0.94 ->
> > 0.98
> > > (or 1.0 if it launches soon). As soon as we can figure this out, we
> will
> > > migrate over.
> > >
> > > On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
> esteban@cloudera.com
> > >
> > > wrote:
> > > >
> > > > Hi Lars,
> > > >
> > > > Thanks for bringing this for discussion. From my experience I can
> tell
> > > that
> > > > 0.94 is very stable but that shouldn't be a blocker to consider to
> > > EOL'ing.
> > > > Are you considering any specific timeframe for that?
> > > >
> > > > thanks,
> > > > esteban.
> > > >
> > > > --
> > > > Cloudera, Inc.
> > > >
> > > >
> > > > On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com>
> > > wrote:
> > > > >
> > > > > given that CDH4 is hbase 0.94 i dont believe nobody is using it.
> for
> > > our
> > > > > clients the majority is on 0.94 (versus 0.96 and up).
> > > > >
> > > > > so i am going with 1), its very stable!
> > > > >
> > > > > On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org>
> > > wrote:
> > > > > >
> > > > > > Over the past few months the rate of the change into 0.94 has
> > slowed
> > > > > > significantly.
> > > > > > 0.94.25 was released on Nov 15th, and since then we had only 4
> > > changes.
> > > > > >
> > > > > > This could mean two things: (1) 0.94 is very stable now or (2)
> > nobody
> > > > is
> > > > > > using it (at least nobody is contributing to it anymore).
> > > > > >
> > > > > >
> > > > > > If anybody out there is still using 0.94 and is not planning to
> > > upgrade
> > > > > to
> > > > > > 0.98 or later soon (which will required downtime), please speak
> up.
> > > > > > Otherwise it might be time to think about EOL'ing 0.94.
> > > > > >
> > > > > > It's not actually much work to do these releases, especially when
> > > they
> > > > > are
> > > > > > so small, but I'd like to continue only if they are actually
> used.
> > > > > > In any case, I am going to spin 0.94.26 with the current 4 fixes
> > > today
> > > > or
> > > > > > tomorrow.
> > > > > >
> > > > > > -- Lars
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: 0.94 going forward
Posted by Sean Busbey <bu...@cloudera.com>.
Does replication and snapshot export work from 0.94.6+ to a 0.96 or 0.98
cluster?
Presuming it does, shouldn't a site be able to use a multiple cluster set
up to do a cut over of a client application?
That doesn't help with needing downtime for to do the eventual upgrade, but
it mitigates the impact on the downstream app.
--
Sean
On Dec 15, 2014 6:51 PM, "Jeremy Carroll" <ph...@gmail.com> wrote:
> Which is why I feel that a lot of customers are still on 0.94. Pretty much
> trapped unless you want to take downtime for your site. Any type of
> guidance would be helpful. We are currently in the process of designing our
> own system to deal with this.
>
> On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> >
> > Zero downtime upgrade from 0.94 won't be possible. See
> > http://hbase.apache.org/book.html#d0e5199
> >
> >
> > On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <ph...@gmail.com>
> > wrote:
> > >
> > > Looking for guidance on how to do a zero downtime upgrade from 0.94 ->
> > 0.98
> > > (or 1.0 if it launches soon). As soon as we can figure this out, we
> will
> > > migrate over.
> > >
> > > On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <
> esteban@cloudera.com
> > >
> > > wrote:
> > > >
> > > > Hi Lars,
> > > >
> > > > Thanks for bringing this for discussion. From my experience I can
> tell
> > > that
> > > > 0.94 is very stable but that shouldn't be a blocker to consider to
> > > EOL'ing.
> > > > Are you considering any specific timeframe for that?
> > > >
> > > > thanks,
> > > > esteban.
> > > >
> > > > --
> > > > Cloudera, Inc.
> > > >
> > > >
> > > > On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com>
> > > wrote:
> > > > >
> > > > > given that CDH4 is hbase 0.94 i dont believe nobody is using it.
> for
> > > our
> > > > > clients the majority is on 0.94 (versus 0.96 and up).
> > > > >
> > > > > so i am going with 1), its very stable!
> > > > >
> > > > > On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org>
> > > wrote:
> > > > > >
> > > > > > Over the past few months the rate of the change into 0.94 has
> > slowed
> > > > > > significantly.
> > > > > > 0.94.25 was released on Nov 15th, and since then we had only 4
> > > changes.
> > > > > >
> > > > > > This could mean two things: (1) 0.94 is very stable now or (2)
> > nobody
> > > > is
> > > > > > using it (at least nobody is contributing to it anymore).
> > > > > >
> > > > > >
> > > > > > If anybody out there is still using 0.94 and is not planning to
> > > upgrade
> > > > > to
> > > > > > 0.98 or later soon (which will required downtime), please speak
> up.
> > > > > > Otherwise it might be time to think about EOL'ing 0.94.
> > > > > >
> > > > > > It's not actually much work to do these releases, especially when
> > > they
> > > > > are
> > > > > > so small, but I'd like to continue only if they are actually
> used.
> > > > > > In any case, I am going to spin 0.94.26 with the current 4 fixes
> > > today
> > > > or
> > > > > > tomorrow.
> > > > > >
> > > > > > -- Lars
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: 0.94 going forward
Posted by Jeremy Carroll <ph...@gmail.com>.
Which is why I feel that a lot of customers are still on 0.94. Pretty much
trapped unless you want to take downtime for your site. Any type of
guidance would be helpful. We are currently in the process of designing our
own system to deal with this.
On Mon, Dec 15, 2014 at 4:47 PM, Andrew Purtell <ap...@apache.org> wrote:
>
> Zero downtime upgrade from 0.94 won't be possible. See
> http://hbase.apache.org/book.html#d0e5199
>
>
> On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <ph...@gmail.com>
> wrote:
> >
> > Looking for guidance on how to do a zero downtime upgrade from 0.94 ->
> 0.98
> > (or 1.0 if it launches soon). As soon as we can figure this out, we will
> > migrate over.
> >
> > On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <esteban@cloudera.com
> >
> > wrote:
> > >
> > > Hi Lars,
> > >
> > > Thanks for bringing this for discussion. From my experience I can tell
> > that
> > > 0.94 is very stable but that shouldn't be a blocker to consider to
> > EOL'ing.
> > > Are you considering any specific timeframe for that?
> > >
> > > thanks,
> > > esteban.
> > >
> > > --
> > > Cloudera, Inc.
> > >
> > >
> > > On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com>
> > wrote:
> > > >
> > > > given that CDH4 is hbase 0.94 i dont believe nobody is using it. for
> > our
> > > > clients the majority is on 0.94 (versus 0.96 and up).
> > > >
> > > > so i am going with 1), its very stable!
> > > >
> > > > On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org>
> > wrote:
> > > > >
> > > > > Over the past few months the rate of the change into 0.94 has
> slowed
> > > > > significantly.
> > > > > 0.94.25 was released on Nov 15th, and since then we had only 4
> > changes.
> > > > >
> > > > > This could mean two things: (1) 0.94 is very stable now or (2)
> nobody
> > > is
> > > > > using it (at least nobody is contributing to it anymore).
> > > > >
> > > > >
> > > > > If anybody out there is still using 0.94 and is not planning to
> > upgrade
> > > > to
> > > > > 0.98 or later soon (which will required downtime), please speak up.
> > > > > Otherwise it might be time to think about EOL'ing 0.94.
> > > > >
> > > > > It's not actually much work to do these releases, especially when
> > they
> > > > are
> > > > > so small, but I'd like to continue only if they are actually used.
> > > > > In any case, I am going to spin 0.94.26 with the current 4 fixes
> > today
> > > or
> > > > > tomorrow.
> > > > >
> > > > > -- Lars
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: 0.94 going forward
Posted by Andrew Purtell <ap...@apache.org>.
Zero downtime upgrade from 0.94 won't be possible. See
http://hbase.apache.org/book.html#d0e5199
On Mon, Dec 15, 2014 at 4:44 PM, Jeremy Carroll <ph...@gmail.com> wrote:
>
> Looking for guidance on how to do a zero downtime upgrade from 0.94 -> 0.98
> (or 1.0 if it launches soon). As soon as we can figure this out, we will
> migrate over.
>
> On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
> >
> > Hi Lars,
> >
> > Thanks for bringing this for discussion. From my experience I can tell
> that
> > 0.94 is very stable but that shouldn't be a blocker to consider to
> EOL'ing.
> > Are you considering any specific timeframe for that?
> >
> > thanks,
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com>
> wrote:
> > >
> > > given that CDH4 is hbase 0.94 i dont believe nobody is using it. for
> our
> > > clients the majority is on 0.94 (versus 0.96 and up).
> > >
> > > so i am going with 1), its very stable!
> > >
> > > On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org>
> wrote:
> > > >
> > > > Over the past few months the rate of the change into 0.94 has slowed
> > > > significantly.
> > > > 0.94.25 was released on Nov 15th, and since then we had only 4
> changes.
> > > >
> > > > This could mean two things: (1) 0.94 is very stable now or (2) nobody
> > is
> > > > using it (at least nobody is contributing to it anymore).
> > > >
> > > >
> > > > If anybody out there is still using 0.94 and is not planning to
> upgrade
> > > to
> > > > 0.98 or later soon (which will required downtime), please speak up.
> > > > Otherwise it might be time to think about EOL'ing 0.94.
> > > >
> > > > It's not actually much work to do these releases, especially when
> they
> > > are
> > > > so small, but I'd like to continue only if they are actually used.
> > > > In any case, I am going to spin 0.94.26 with the current 4 fixes
> today
> > or
> > > > tomorrow.
> > > >
> > > > -- Lars
> > > >
> > >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: 0.94 going forward
Posted by Jeremy Carroll <ph...@gmail.com>.
Looking for guidance on how to do a zero downtime upgrade from 0.94 -> 0.98
(or 1.0 if it launches soon). As soon as we can figure this out, we will
migrate over.
On Mon, Dec 15, 2014 at 1:37 PM, Esteban Gutierrez <es...@cloudera.com>
wrote:
>
> Hi Lars,
>
> Thanks for bringing this for discussion. From my experience I can tell that
> 0.94 is very stable but that shouldn't be a blocker to consider to EOL'ing.
> Are you considering any specific timeframe for that?
>
> thanks,
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com> wrote:
> >
> > given that CDH4 is hbase 0.94 i dont believe nobody is using it. for our
> > clients the majority is on 0.94 (versus 0.96 and up).
> >
> > so i am going with 1), its very stable!
> >
> > On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org> wrote:
> > >
> > > Over the past few months the rate of the change into 0.94 has slowed
> > > significantly.
> > > 0.94.25 was released on Nov 15th, and since then we had only 4 changes.
> > >
> > > This could mean two things: (1) 0.94 is very stable now or (2) nobody
> is
> > > using it (at least nobody is contributing to it anymore).
> > >
> > >
> > > If anybody out there is still using 0.94 and is not planning to upgrade
> > to
> > > 0.98 or later soon (which will required downtime), please speak up.
> > > Otherwise it might be time to think about EOL'ing 0.94.
> > >
> > > It's not actually much work to do these releases, especially when they
> > are
> > > so small, but I'd like to continue only if they are actually used.
> > > In any case, I am going to spin 0.94.26 with the current 4 fixes today
> or
> > > tomorrow.
> > >
> > > -- Lars
> > >
> >
>
Re: 0.94 going forward
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hi Lars,
Thanks for bringing this for discussion. From my experience I can tell that
0.94 is very stable but that shouldn't be a blocker to consider to EOL'ing.
Are you considering any specific timeframe for that?
thanks,
esteban.
--
Cloudera, Inc.
On Mon, Dec 15, 2014 at 11:46 AM, Koert Kuipers <ko...@tresata.com> wrote:
>
> given that CDH4 is hbase 0.94 i dont believe nobody is using it. for our
> clients the majority is on 0.94 (versus 0.96 and up).
>
> so i am going with 1), its very stable!
>
> On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org> wrote:
> >
> > Over the past few months the rate of the change into 0.94 has slowed
> > significantly.
> > 0.94.25 was released on Nov 15th, and since then we had only 4 changes.
> >
> > This could mean two things: (1) 0.94 is very stable now or (2) nobody is
> > using it (at least nobody is contributing to it anymore).
> >
> >
> > If anybody out there is still using 0.94 and is not planning to upgrade
> to
> > 0.98 or later soon (which will required downtime), please speak up.
> > Otherwise it might be time to think about EOL'ing 0.94.
> >
> > It's not actually much work to do these releases, especially when they
> are
> > so small, but I'd like to continue only if they are actually used.
> > In any case, I am going to spin 0.94.26 with the current 4 fixes today or
> > tomorrow.
> >
> > -- Lars
> >
>
Re: 0.94 going forward
Posted by Koert Kuipers <ko...@tresata.com>.
given that CDH4 is hbase 0.94 i dont believe nobody is using it. for our
clients the majority is on 0.94 (versus 0.96 and up).
so i am going with 1), its very stable!
On Mon, Dec 15, 2014 at 1:53 PM, lars hofhansl <la...@apache.org> wrote:
>
> Over the past few months the rate of the change into 0.94 has slowed
> significantly.
> 0.94.25 was released on Nov 15th, and since then we had only 4 changes.
>
> This could mean two things: (1) 0.94 is very stable now or (2) nobody is
> using it (at least nobody is contributing to it anymore).
>
>
> If anybody out there is still using 0.94 and is not planning to upgrade to
> 0.98 or later soon (which will required downtime), please speak up.
> Otherwise it might be time to think about EOL'ing 0.94.
>
> It's not actually much work to do these releases, especially when they are
> so small, but I'd like to continue only if they are actually used.
> In any case, I am going to spin 0.94.26 with the current 4 fixes today or
> tomorrow.
>
> -- Lars
>