You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by Sean Busbey <bu...@cloudera.com> on 2014/07/11 21:22:05 UTC
Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
through 1.5?
An initial port of the 1.4 -> 1.6 upgrade code for the current
1.6.1-SNAPSHOT branch is now up:
* https://issues.apache.org/jira/browse/ACCUMULO-2988
* https://reviews.apache.org/r/23413/
On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org> wrote:
> Cool. Thanks, Sean!
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > I'll be creating a ticket and posting a patch this week.
> >
> >
> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org>
> wrote:
> >
> > > So, just to revisit this conversation, it seems like there is interest
> in
> > > supporting this. Is there already a ticket for it and/or somebody
> > > interested in doing the necessary work for 1.6.1?
> > >
> > >
> > > --
> > > Christopher L Tubbs II
> > > http://gravatar.com/ctubbsii
> > >
> > >
> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com>
> wrote:
> > >
> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > > > (ACCUMULO-2770), start 1.6
> > > >
> > > > Mike
> > > >
> > > >
> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <dr...@gmail.com>
> > > > wrote:
> > > >
> > > > > Mike,
> > > > >
> > > > > So works just like upgrading from 1.5?
> > > > >
> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > > > >
> > > > > That sounds entirely reasonable.
> > > > >
> > > > > Drew
> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com> wrote:
> > > > >
> > > > > > We initially tried to set it up as a stand-alone utility but
> > > eventually
> > > > > > gave up. In order to properly do the upgrade, you concurrently
> need
> > > to
> > > > > run
> > > > > > whatever upgrade code concurrently with a tablet server hosting
> > > > !METADATA
> > > > > > and a tablet server that can replay WALs. We ended up
> duplicating a
> > > lot
> > > > > of
> > > > > > logic already present in master before scrapping that plan. An
> > > > > alternative
> > > > > > would have been to try to build on MAC, but that was also
> > non-trivial
> > > > to
> > > > > > deploy, so we spliced the code into the existing upgrade path.
> How
> > do
> > > > you
> > > > > > feel about that, Drew?
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > drew.farris@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I'm +1 for a utility that would allow us to go directly from
> 1.4
> > to
> > > > > 1.6.
> > > > > > >
> > > > > > > In terms of a general policy, I suggest we make this sort of
> > > decision
> > > > > on
> > > > > > a
> > > > > > > case by case basis. My unreasonably self-centered intuition
> > > suggests
> > > > > that
> > > > > > > there may be some folks that want to go from 1.4 to 1.6 now due
> > to
> > > a
> > > > > > > relatively short 1.5 cycle. The need to jump multiple versions
> > like
> > > > > might
> > > > > > > not exist in the future.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > busbey@cloudera.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > In an effort to get more users off of our now unsupported 1.4
> > > > > release,
> > > > > > > > should we support upgrading directly to 1.6 without going
> > > through a
> > > > > 1.5
> > > > > > > > upgrade?
> > > > > > > >
> > > > > > > > More directly for those on user@: would you be more likely
> to
> > > > > upgrade
> > > > > > > off
> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > > > > > > >
> > > > > > > > We have this working locally at Cloudera as a part of our CDH
> > > > > > integration
> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > > > > > >
> > > > > > > > We can get into implementation details on a jira if there's
> > > > positive
> > > > > > > > consensus, but the changes weren't very complicated. They're
> > > mostly
> > > > > > > >
> > > > > > > > * forward porting and consolidating some upgrade code
> > > > > > > > * additions to the README for instructions
> > > > > > > >
> > > > > > > > Personally, I can see the both sides of the argument. On the
> > plus
> > > > > side,
> > > > > > > > anything to get more users off of 1.4 is a good thing. On the
> > > > > negative
> > > > > > > > side, it means we have the 1.4 related upgrade code sitting
> in
> > a
> > > > > > > supported
> > > > > > > > code branch longer.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > --
> > > > > > > > Sean
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sean
> >
>
--
Sean
Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
through 1.5?
Posted by Sean Busbey <bu...@cloudera.com>.
On Wed, Jul 30, 2014 at 5:55 PM, <dl...@comcast.net> wrote:
> Please add a check that instance.dfs.uri, although deprecated, is
> configured in the upgrade and that instance.volumes is NOT configured, see
> the discussion in ACCUMULO-3006.
>
>
Dave, how is that change related to the changes in ACCUMULO-2988?
I'm all for adding that check. However, it's something we should be doing
for all 1.6 upgrades; it isn't a part of adding teh 1.4 -> 1.6 path.
-Sean
> -----Original Message-----
> From: Mike Drob [mailto:madrob@cloudera.com]
> Sent: Wednesday, July 30, 2014 6:12 PM
> To: Accumulo Dev List
> Subject: Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
> through 1.5?
>
> The upgrade test looks something like:
>
> * Start up 1.4 (or 1.5)
> * Create tables with a variety of configurations (LZO/Snappy/GZ, Bloom
> Filter On/Off, Block Cache On/Off)
> * Load some data into these tables, enough such that flushes and
> compactions start to occur.
> * Abruptly kill all of the servers. This ensures that there are WALs
> around. Unfortunately, we can't have any compactions in progress at this
> point, since that causes a Fate Operation, and that would prevent the
> upgrade from completing.
> * Upgrade the bits, start 1.6, and wait for the upgrade steps to finish.
> * Check for data loss
> * Trigger some compactions and wait for those to finish. In a previous
> iteration, we discovered an issue where the relative path'd files and WALs
> were not being properly deleted from !0.
> * Check for data loss.
> * Restart the tablet servers to force tablets to reload (test for orphaned
> files).
> * Check for data loss.
>
> We did not try using custom iterators.
>
> Mike
>
>
> On Wed, Jul 30, 2014 at 4:58 PM, Christopher <ct...@apache.org> wrote:
>
> > Do you have a list of test cases you've tried?
> > (like with/without walogs, custom iterators on tables, delete markers
> > for the metadata table, etc.)?
> >
> > If we could consolidate some of those into a comprehensive list,
> > that'd be pretty useful, I'd think, to give confidence that we didn't
> > miss anything in the review. It might also help focus testing for
> > 1.6.1 release after you've pushed.
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> >
> > On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
> > >
> > > https://reviews.apache.org/r/23413/
> > >
> > > Pending any additional concerns, I'll be pushing this soon.
> > >
> > >
> > > On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > > > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > > > 1.6.1-SNAPSHOT branch is now up:
> > > >
> > > > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > > > * https://reviews.apache.org/r/23413/
> > > >
> > > >
> > > > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> > > wrote:
> > > >
> > > >> Cool. Thanks, Sean!
> > > >>
> > > >>
> > > >> --
> > > >> Christopher L Tubbs II
> > > >> http://gravatar.com/ctubbsii
> > > >>
> > > >>
> > > >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey
> > > >> <bu...@cloudera.com>
> > > wrote:
> > > >>
> > > >> > I'll be creating a ticket and posting a patch this week.
> > > >> >
> > > >> >
> > > >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher
> > > >> > <ct...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > So, just to revisit this conversation, it seems like there is
> > > >> interest in
> > > >> > > supporting this. Is there already a ticket for it and/or
> > > >> > > somebody interested in doing the necessary work for 1.6.1?
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Christopher L Tubbs II
> > > >> > > http://gravatar.com/ctubbsii
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob
> > > >> > > <ma...@cloudera.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > > >> > > > (ACCUMULO-2770), start 1.6
> > > >> > > >
> > > >> > > > Mike
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> > > drew.farris@gmail.com
> > > >> >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Mike,
> > > >> > > > >
> > > >> > > > > So works just like upgrading from 1.5?
> > > >> > > > >
> > > >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > > >> > > > >
> > > >> > > > > That sounds entirely reasonable.
> > > >> > > > >
> > > >> > > > > Drew
> > > >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob"
> > > >> > > > > <ma...@cloudera.com>
> > > >> wrote:
> > > >> > > > >
> > > >> > > > > > We initially tried to set it up as a stand-alone
> > > >> > > > > > utility but
> > > >> > > eventually
> > > >> > > > > > gave up. In order to properly do the upgrade, you
> > concurrently
> > > >> need
> > > >> > > to
> > > >> > > > > run
> > > >> > > > > > whatever upgrade code concurrently with a tablet server
> > > hosting
> > > >> > > > !METADATA
> > > >> > > > > > and a tablet server that can replay WALs. We ended up
> > > >> duplicating a
> > > >> > > lot
> > > >> > > > > of
> > > >> > > > > > logic already present in master before scrapping that
> plan.
> > An
> > > >> > > > > alternative
> > > >> > > > > > would have been to try to build on MAC, but that was
> > > >> > > > > > also
> > > >> > non-trivial
> > > >> > > > to
> > > >> > > > > > deploy, so we spliced the code into the existing
> > > >> > > > > > upgrade
> > path.
> > > >> How
> > > >> > do
> > > >> > > > you
> > > >> > > > > > feel about that, Drew?
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > > >> > drew.farris@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > I'm +1 for a utility that would allow us to go
> > > >> > > > > > > directly
> > from
> > > >> 1.4
> > > >> > to
> > > >> > > > > 1.6.
> > > >> > > > > > >
> > > >> > > > > > > In terms of a general policy, I suggest we make this
> > > >> > > > > > > sort
> > of
> > > >> > > decision
> > > >> > > > > on
> > > >> > > > > > a
> > > >> > > > > > > case by case basis. My unreasonably self-centered
> > intuition
> > > >> > > suggests
> > > >> > > > > that
> > > >> > > > > > > there may be some folks that want to go from 1.4 to
> > > >> > > > > > > 1.6
> > now
> > > >> due
> > > >> > to
> > > >> > > a
> > > >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> > > versions
> > > >> > like
> > > >> > > > > might
> > > >> > > > > > > not exist in the future.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > > >> > busbey@cloudera.com>
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > In an effort to get more users off of our now
> > unsupported
> > > >> 1.4
> > > >> > > > > release,
> > > >> > > > > > > > should we support upgrading directly to 1.6 without
> > going
> > > >> > > through a
> > > >> > > > > 1.5
> > > >> > > > > > > > upgrade?
> > > >> > > > > > > >
> > > >> > > > > > > > More directly for those on user@: would you be more
> > > likely
> > > >> to
> > > >> > > > > upgrade
> > > >> > > > > > > off
> > > >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > > >> > > > > > > >
> > > >> > > > > > > > We have this working locally at Cloudera as a part
> > > >> > > > > > > > of
> > our
> > > >> CDH
> > > >> > > > > > integration
> > > >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > >> > > > > > > >
> > > >> > > > > > > > We can get into implementation details on a jira if
> > > there's
> > > >> > > > positive
> > > >> > > > > > > > consensus, but the changes weren't very complicated.
> > > They're
> > > >> > > mostly
> > > >> > > > > > > >
> > > >> > > > > > > > * forward porting and consolidating some upgrade
> > > >> > > > > > > > code
> > > >> > > > > > > > * additions to the README for instructions
> > > >> > > > > > > >
> > > >> > > > > > > > Personally, I can see the both sides of the
> > > >> > > > > > > > argument. On
> > > the
> > > >> > plus
> > > >> > > > > side,
> > > >> > > > > > > > anything to get more users off of 1.4 is a good thing.
> > On
> > > >> the
> > > >> > > > > negative
> > > >> > > > > > > > side, it means we have the 1.4 related upgrade code
> > > sitting
> > > >> in
> > > >> > a
> > > >> > > > > > > supported
> > > >> > > > > > > > code branch longer.
> > > >> > > > > > > >
> > > >> > > > > > > > Thoughts?
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > > Sean
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Sean
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
>
>
--
Sean
Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
through 1.5?
Posted by Mike Drob <ma...@cloudera.com>.
That seems out of scope for this issue. Can it be done as part of the work
on ACCUMULO-3006 when that gets in?
On Wed, Jul 30, 2014 at 5:55 PM, <dl...@comcast.net> wrote:
> Please add a check that instance.dfs.uri, although deprecated, is
> configured in the upgrade and that instance.volumes is NOT configured, see
> the discussion in ACCUMULO-3006.
>
> -----Original Message-----
> From: Mike Drob [mailto:madrob@cloudera.com]
> Sent: Wednesday, July 30, 2014 6:12 PM
> To: Accumulo Dev List
> Subject: Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
> through 1.5?
>
> The upgrade test looks something like:
>
> * Start up 1.4 (or 1.5)
> * Create tables with a variety of configurations (LZO/Snappy/GZ, Bloom
> Filter On/Off, Block Cache On/Off)
> * Load some data into these tables, enough such that flushes and
> compactions start to occur.
> * Abruptly kill all of the servers. This ensures that there are WALs
> around. Unfortunately, we can't have any compactions in progress at this
> point, since that causes a Fate Operation, and that would prevent the
> upgrade from completing.
> * Upgrade the bits, start 1.6, and wait for the upgrade steps to finish.
> * Check for data loss
> * Trigger some compactions and wait for those to finish. In a previous
> iteration, we discovered an issue where the relative path'd files and WALs
> were not being properly deleted from !0.
> * Check for data loss.
> * Restart the tablet servers to force tablets to reload (test for orphaned
> files).
> * Check for data loss.
>
> We did not try using custom iterators.
>
> Mike
>
>
> On Wed, Jul 30, 2014 at 4:58 PM, Christopher <ct...@apache.org> wrote:
>
> > Do you have a list of test cases you've tried?
> > (like with/without walogs, custom iterators on tables, delete markers
> > for the metadata table, etc.)?
> >
> > If we could consolidate some of those into a comprehensive list,
> > that'd be pretty useful, I'd think, to give confidence that we didn't
> > miss anything in the review. It might also help focus testing for
> > 1.6.1 release after you've pushed.
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> >
> > On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
> > >
> > > https://reviews.apache.org/r/23413/
> > >
> > > Pending any additional concerns, I'll be pushing this soon.
> > >
> > >
> > > On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > > > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > > > 1.6.1-SNAPSHOT branch is now up:
> > > >
> > > > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > > > * https://reviews.apache.org/r/23413/
> > > >
> > > >
> > > > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> > > wrote:
> > > >
> > > >> Cool. Thanks, Sean!
> > > >>
> > > >>
> > > >> --
> > > >> Christopher L Tubbs II
> > > >> http://gravatar.com/ctubbsii
> > > >>
> > > >>
> > > >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey
> > > >> <bu...@cloudera.com>
> > > wrote:
> > > >>
> > > >> > I'll be creating a ticket and posting a patch this week.
> > > >> >
> > > >> >
> > > >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher
> > > >> > <ct...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > So, just to revisit this conversation, it seems like there is
> > > >> interest in
> > > >> > > supporting this. Is there already a ticket for it and/or
> > > >> > > somebody interested in doing the necessary work for 1.6.1?
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Christopher L Tubbs II
> > > >> > > http://gravatar.com/ctubbsii
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob
> > > >> > > <ma...@cloudera.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > > >> > > > (ACCUMULO-2770), start 1.6
> > > >> > > >
> > > >> > > > Mike
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> > > drew.farris@gmail.com
> > > >> >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Mike,
> > > >> > > > >
> > > >> > > > > So works just like upgrading from 1.5?
> > > >> > > > >
> > > >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > > >> > > > >
> > > >> > > > > That sounds entirely reasonable.
> > > >> > > > >
> > > >> > > > > Drew
> > > >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob"
> > > >> > > > > <ma...@cloudera.com>
> > > >> wrote:
> > > >> > > > >
> > > >> > > > > > We initially tried to set it up as a stand-alone
> > > >> > > > > > utility but
> > > >> > > eventually
> > > >> > > > > > gave up. In order to properly do the upgrade, you
> > concurrently
> > > >> need
> > > >> > > to
> > > >> > > > > run
> > > >> > > > > > whatever upgrade code concurrently with a tablet server
> > > hosting
> > > >> > > > !METADATA
> > > >> > > > > > and a tablet server that can replay WALs. We ended up
> > > >> duplicating a
> > > >> > > lot
> > > >> > > > > of
> > > >> > > > > > logic already present in master before scrapping that
> plan.
> > An
> > > >> > > > > alternative
> > > >> > > > > > would have been to try to build on MAC, but that was
> > > >> > > > > > also
> > > >> > non-trivial
> > > >> > > > to
> > > >> > > > > > deploy, so we spliced the code into the existing
> > > >> > > > > > upgrade
> > path.
> > > >> How
> > > >> > do
> > > >> > > > you
> > > >> > > > > > feel about that, Drew?
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > > >> > drew.farris@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > I'm +1 for a utility that would allow us to go
> > > >> > > > > > > directly
> > from
> > > >> 1.4
> > > >> > to
> > > >> > > > > 1.6.
> > > >> > > > > > >
> > > >> > > > > > > In terms of a general policy, I suggest we make this
> > > >> > > > > > > sort
> > of
> > > >> > > decision
> > > >> > > > > on
> > > >> > > > > > a
> > > >> > > > > > > case by case basis. My unreasonably self-centered
> > intuition
> > > >> > > suggests
> > > >> > > > > that
> > > >> > > > > > > there may be some folks that want to go from 1.4 to
> > > >> > > > > > > 1.6
> > now
> > > >> due
> > > >> > to
> > > >> > > a
> > > >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> > > versions
> > > >> > like
> > > >> > > > > might
> > > >> > > > > > > not exist in the future.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > > >> > busbey@cloudera.com>
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > In an effort to get more users off of our now
> > unsupported
> > > >> 1.4
> > > >> > > > > release,
> > > >> > > > > > > > should we support upgrading directly to 1.6 without
> > going
> > > >> > > through a
> > > >> > > > > 1.5
> > > >> > > > > > > > upgrade?
> > > >> > > > > > > >
> > > >> > > > > > > > More directly for those on user@: would you be more
> > > likely
> > > >> to
> > > >> > > > > upgrade
> > > >> > > > > > > off
> > > >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > > >> > > > > > > >
> > > >> > > > > > > > We have this working locally at Cloudera as a part
> > > >> > > > > > > > of
> > our
> > > >> CDH
> > > >> > > > > > integration
> > > >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > >> > > > > > > >
> > > >> > > > > > > > We can get into implementation details on a jira if
> > > there's
> > > >> > > > positive
> > > >> > > > > > > > consensus, but the changes weren't very complicated.
> > > They're
> > > >> > > mostly
> > > >> > > > > > > >
> > > >> > > > > > > > * forward porting and consolidating some upgrade
> > > >> > > > > > > > code
> > > >> > > > > > > > * additions to the README for instructions
> > > >> > > > > > > >
> > > >> > > > > > > > Personally, I can see the both sides of the
> > > >> > > > > > > > argument. On
> > > the
> > > >> > plus
> > > >> > > > > side,
> > > >> > > > > > > > anything to get more users off of 1.4 is a good thing.
> > On
> > > >> the
> > > >> > > > > negative
> > > >> > > > > > > > side, it means we have the 1.4 related upgrade code
> > > sitting
> > > >> in
> > > >> > a
> > > >> > > > > > > supported
> > > >> > > > > > > > code branch longer.
> > > >> > > > > > > >
> > > >> > > > > > > > Thoughts?
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > > Sean
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Sean
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
>
>
RE: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?
Posted by dl...@comcast.net.
Please add a check that instance.dfs.uri, although deprecated, is configured in the upgrade and that instance.volumes is NOT configured, see the discussion in ACCUMULO-3006.
-----Original Message-----
From: Mike Drob [mailto:madrob@cloudera.com]
Sent: Wednesday, July 30, 2014 6:12 PM
To: Accumulo Dev List
Subject: Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?
The upgrade test looks something like:
* Start up 1.4 (or 1.5)
* Create tables with a variety of configurations (LZO/Snappy/GZ, Bloom Filter On/Off, Block Cache On/Off)
* Load some data into these tables, enough such that flushes and compactions start to occur.
* Abruptly kill all of the servers. This ensures that there are WALs around. Unfortunately, we can't have any compactions in progress at this point, since that causes a Fate Operation, and that would prevent the upgrade from completing.
* Upgrade the bits, start 1.6, and wait for the upgrade steps to finish.
* Check for data loss
* Trigger some compactions and wait for those to finish. In a previous iteration, we discovered an issue where the relative path'd files and WALs were not being properly deleted from !0.
* Check for data loss.
* Restart the tablet servers to force tablets to reload (test for orphaned files).
* Check for data loss.
We did not try using custom iterators.
Mike
On Wed, Jul 30, 2014 at 4:58 PM, Christopher <ct...@apache.org> wrote:
> Do you have a list of test cases you've tried?
> (like with/without walogs, custom iterators on tables, delete markers
> for the metadata table, etc.)?
>
> If we could consolidate some of those into a comprehensive list,
> that'd be pretty useful, I'd think, to give confidence that we didn't
> miss anything in the review. It might also help focus testing for
> 1.6.1 release after you've pushed.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
> >
> > https://reviews.apache.org/r/23413/
> >
> > Pending any additional concerns, I'll be pushing this soon.
> >
> >
> > On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > > 1.6.1-SNAPSHOT branch is now up:
> > >
> > > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > > * https://reviews.apache.org/r/23413/
> > >
> > >
> > > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> > wrote:
> > >
> > >> Cool. Thanks, Sean!
> > >>
> > >>
> > >> --
> > >> Christopher L Tubbs II
> > >> http://gravatar.com/ctubbsii
> > >>
> > >>
> > >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey
> > >> <bu...@cloudera.com>
> > wrote:
> > >>
> > >> > I'll be creating a ticket and posting a patch this week.
> > >> >
> > >> >
> > >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher
> > >> > <ct...@apache.org>
> > >> wrote:
> > >> >
> > >> > > So, just to revisit this conversation, it seems like there is
> > >> interest in
> > >> > > supporting this. Is there already a ticket for it and/or
> > >> > > somebody interested in doing the necessary work for 1.6.1?
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Christopher L Tubbs II
> > >> > > http://gravatar.com/ctubbsii
> > >> > >
> > >> > >
> > >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob
> > >> > > <ma...@cloudera.com>
> > >> wrote:
> > >> > >
> > >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > >> > > > (ACCUMULO-2770), start 1.6
> > >> > > >
> > >> > > > Mike
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> > drew.farris@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Mike,
> > >> > > > >
> > >> > > > > So works just like upgrading from 1.5?
> > >> > > > >
> > >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > >> > > > >
> > >> > > > > That sounds entirely reasonable.
> > >> > > > >
> > >> > > > > Drew
> > >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob"
> > >> > > > > <ma...@cloudera.com>
> > >> wrote:
> > >> > > > >
> > >> > > > > > We initially tried to set it up as a stand-alone
> > >> > > > > > utility but
> > >> > > eventually
> > >> > > > > > gave up. In order to properly do the upgrade, you
> concurrently
> > >> need
> > >> > > to
> > >> > > > > run
> > >> > > > > > whatever upgrade code concurrently with a tablet server
> > hosting
> > >> > > > !METADATA
> > >> > > > > > and a tablet server that can replay WALs. We ended up
> > >> duplicating a
> > >> > > lot
> > >> > > > > of
> > >> > > > > > logic already present in master before scrapping that plan.
> An
> > >> > > > > alternative
> > >> > > > > > would have been to try to build on MAC, but that was
> > >> > > > > > also
> > >> > non-trivial
> > >> > > > to
> > >> > > > > > deploy, so we spliced the code into the existing
> > >> > > > > > upgrade
> path.
> > >> How
> > >> > do
> > >> > > > you
> > >> > > > > > feel about that, Drew?
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > >> > drew.farris@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > I'm +1 for a utility that would allow us to go
> > >> > > > > > > directly
> from
> > >> 1.4
> > >> > to
> > >> > > > > 1.6.
> > >> > > > > > >
> > >> > > > > > > In terms of a general policy, I suggest we make this
> > >> > > > > > > sort
> of
> > >> > > decision
> > >> > > > > on
> > >> > > > > > a
> > >> > > > > > > case by case basis. My unreasonably self-centered
> intuition
> > >> > > suggests
> > >> > > > > that
> > >> > > > > > > there may be some folks that want to go from 1.4 to
> > >> > > > > > > 1.6
> now
> > >> due
> > >> > to
> > >> > > a
> > >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> > versions
> > >> > like
> > >> > > > > might
> > >> > > > > > > not exist in the future.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > >> > busbey@cloudera.com>
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > In an effort to get more users off of our now
> unsupported
> > >> 1.4
> > >> > > > > release,
> > >> > > > > > > > should we support upgrading directly to 1.6 without
> going
> > >> > > through a
> > >> > > > > 1.5
> > >> > > > > > > > upgrade?
> > >> > > > > > > >
> > >> > > > > > > > More directly for those on user@: would you be more
> > likely
> > >> to
> > >> > > > > upgrade
> > >> > > > > > > off
> > >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > >> > > > > > > >
> > >> > > > > > > > We have this working locally at Cloudera as a part
> > >> > > > > > > > of
> our
> > >> CDH
> > >> > > > > > integration
> > >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > >> > > > > > > >
> > >> > > > > > > > We can get into implementation details on a jira if
> > there's
> > >> > > > positive
> > >> > > > > > > > consensus, but the changes weren't very complicated.
> > They're
> > >> > > mostly
> > >> > > > > > > >
> > >> > > > > > > > * forward porting and consolidating some upgrade
> > >> > > > > > > > code
> > >> > > > > > > > * additions to the README for instructions
> > >> > > > > > > >
> > >> > > > > > > > Personally, I can see the both sides of the
> > >> > > > > > > > argument. On
> > the
> > >> > plus
> > >> > > > > side,
> > >> > > > > > > > anything to get more users off of 1.4 is a good thing.
> On
> > >> the
> > >> > > > > negative
> > >> > > > > > > > side, it means we have the 1.4 related upgrade code
> > sitting
> > >> in
> > >> > a
> > >> > > > > > > supported
> > >> > > > > > > > code branch longer.
> > >> > > > > > > >
> > >> > > > > > > > Thoughts?
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > > Sean
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Sean
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
> >
> >
> > --
> > Sean
> >
>
Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
through 1.5?
Posted by Mike Drob <ma...@cloudera.com>.
The upgrade test looks something like:
* Start up 1.4 (or 1.5)
* Create tables with a variety of configurations (LZO/Snappy/GZ, Bloom
Filter On/Off, Block Cache On/Off)
* Load some data into these tables, enough such that flushes and
compactions start to occur.
* Abruptly kill all of the servers. This ensures that there are WALs
around. Unfortunately, we can't have any compactions in progress at this
point, since that causes a Fate Operation, and that would prevent the
upgrade from completing.
* Upgrade the bits, start 1.6, and wait for the upgrade steps to finish.
* Check for data loss
* Trigger some compactions and wait for those to finish. In a previous
iteration, we discovered an issue where the relative path'd files and WALs
were not being properly deleted from !0.
* Check for data loss.
* Restart the tablet servers to force tablets to reload (test for orphaned
files).
* Check for data loss.
We did not try using custom iterators.
Mike
On Wed, Jul 30, 2014 at 4:58 PM, Christopher <ct...@apache.org> wrote:
> Do you have a list of test cases you've tried?
> (like with/without walogs, custom iterators on tables, delete markers for
> the metadata table, etc.)?
>
> If we could consolidate some of those into a comprehensive list, that'd be
> pretty useful, I'd think, to give confidence that we didn't miss anything
> in the review. It might also help focus testing for 1.6.1 release after
> you've pushed.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
> >
> > https://reviews.apache.org/r/23413/
> >
> > Pending any additional concerns, I'll be pushing this soon.
> >
> >
> > On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > > 1.6.1-SNAPSHOT branch is now up:
> > >
> > > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > > * https://reviews.apache.org/r/23413/
> > >
> > >
> > > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> > wrote:
> > >
> > >> Cool. Thanks, Sean!
> > >>
> > >>
> > >> --
> > >> Christopher L Tubbs II
> > >> http://gravatar.com/ctubbsii
> > >>
> > >>
> > >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >>
> > >> > I'll be creating a ticket and posting a patch this week.
> > >> >
> > >> >
> > >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org>
> > >> wrote:
> > >> >
> > >> > > So, just to revisit this conversation, it seems like there is
> > >> interest in
> > >> > > supporting this. Is there already a ticket for it and/or somebody
> > >> > > interested in doing the necessary work for 1.6.1?
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Christopher L Tubbs II
> > >> > > http://gravatar.com/ctubbsii
> > >> > >
> > >> > >
> > >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com>
> > >> wrote:
> > >> > >
> > >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > >> > > > (ACCUMULO-2770), start 1.6
> > >> > > >
> > >> > > > Mike
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> > drew.farris@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Mike,
> > >> > > > >
> > >> > > > > So works just like upgrading from 1.5?
> > >> > > > >
> > >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > >> > > > >
> > >> > > > > That sounds entirely reasonable.
> > >> > > > >
> > >> > > > > Drew
> > >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com>
> > >> wrote:
> > >> > > > >
> > >> > > > > > We initially tried to set it up as a stand-alone utility but
> > >> > > eventually
> > >> > > > > > gave up. In order to properly do the upgrade, you
> concurrently
> > >> need
> > >> > > to
> > >> > > > > run
> > >> > > > > > whatever upgrade code concurrently with a tablet server
> > hosting
> > >> > > > !METADATA
> > >> > > > > > and a tablet server that can replay WALs. We ended up
> > >> duplicating a
> > >> > > lot
> > >> > > > > of
> > >> > > > > > logic already present in master before scrapping that plan.
> An
> > >> > > > > alternative
> > >> > > > > > would have been to try to build on MAC, but that was also
> > >> > non-trivial
> > >> > > > to
> > >> > > > > > deploy, so we spliced the code into the existing upgrade
> path.
> > >> How
> > >> > do
> > >> > > > you
> > >> > > > > > feel about that, Drew?
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > >> > drew.farris@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > I'm +1 for a utility that would allow us to go directly
> from
> > >> 1.4
> > >> > to
> > >> > > > > 1.6.
> > >> > > > > > >
> > >> > > > > > > In terms of a general policy, I suggest we make this sort
> of
> > >> > > decision
> > >> > > > > on
> > >> > > > > > a
> > >> > > > > > > case by case basis. My unreasonably self-centered
> intuition
> > >> > > suggests
> > >> > > > > that
> > >> > > > > > > there may be some folks that want to go from 1.4 to 1.6
> now
> > >> due
> > >> > to
> > >> > > a
> > >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> > versions
> > >> > like
> > >> > > > > might
> > >> > > > > > > not exist in the future.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > >> > busbey@cloudera.com>
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > In an effort to get more users off of our now
> unsupported
> > >> 1.4
> > >> > > > > release,
> > >> > > > > > > > should we support upgrading directly to 1.6 without
> going
> > >> > > through a
> > >> > > > > 1.5
> > >> > > > > > > > upgrade?
> > >> > > > > > > >
> > >> > > > > > > > More directly for those on user@: would you be more
> > likely
> > >> to
> > >> > > > > upgrade
> > >> > > > > > > off
> > >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > >> > > > > > > >
> > >> > > > > > > > We have this working locally at Cloudera as a part of
> our
> > >> CDH
> > >> > > > > > integration
> > >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > >> > > > > > > >
> > >> > > > > > > > We can get into implementation details on a jira if
> > there's
> > >> > > > positive
> > >> > > > > > > > consensus, but the changes weren't very complicated.
> > They're
> > >> > > mostly
> > >> > > > > > > >
> > >> > > > > > > > * forward porting and consolidating some upgrade code
> > >> > > > > > > > * additions to the README for instructions
> > >> > > > > > > >
> > >> > > > > > > > Personally, I can see the both sides of the argument. On
> > the
> > >> > plus
> > >> > > > > side,
> > >> > > > > > > > anything to get more users off of 1.4 is a good thing.
> On
> > >> the
> > >> > > > > negative
> > >> > > > > > > > side, it means we have the 1.4 related upgrade code
> > sitting
> > >> in
> > >> > a
> > >> > > > > > > supported
> > >> > > > > > > > code branch longer.
> > >> > > > > > > >
> > >> > > > > > > > Thoughts?
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > > Sean
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Sean
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
> >
> >
> > --
> > Sean
> >
>
Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
through 1.5?
Posted by Christopher <ct...@apache.org>.
Do you have a list of test cases you've tried?
(like with/without walogs, custom iterators on tables, delete markers for
the metadata table, etc.)?
If we could consolidate some of those into a comprehensive list, that'd be
pretty useful, I'd think, to give confidence that we didn't miss anything
in the review. It might also help focus testing for 1.6.1 release after
you've pushed.
--
Christopher L Tubbs II
http://gravatar.com/ctubbsii
On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com> wrote:
> Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
>
> https://reviews.apache.org/r/23413/
>
> Pending any additional concerns, I'll be pushing this soon.
>
>
> On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > 1.6.1-SNAPSHOT branch is now up:
> >
> > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > * https://reviews.apache.org/r/23413/
> >
> >
> > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> wrote:
> >
> >> Cool. Thanks, Sean!
> >>
> >>
> >> --
> >> Christopher L Tubbs II
> >> http://gravatar.com/ctubbsii
> >>
> >>
> >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >>
> >> > I'll be creating a ticket and posting a patch this week.
> >> >
> >> >
> >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org>
> >> wrote:
> >> >
> >> > > So, just to revisit this conversation, it seems like there is
> >> interest in
> >> > > supporting this. Is there already a ticket for it and/or somebody
> >> > > interested in doing the necessary work for 1.6.1?
> >> > >
> >> > >
> >> > > --
> >> > > Christopher L Tubbs II
> >> > > http://gravatar.com/ctubbsii
> >> > >
> >> > >
> >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com>
> >> wrote:
> >> > >
> >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> >> > > > (ACCUMULO-2770), start 1.6
> >> > > >
> >> > > > Mike
> >> > > >
> >> > > >
> >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> drew.farris@gmail.com
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > Mike,
> >> > > > >
> >> > > > > So works just like upgrading from 1.5?
> >> > > > >
> >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> >> > > > >
> >> > > > > That sounds entirely reasonable.
> >> > > > >
> >> > > > > Drew
> >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com>
> >> wrote:
> >> > > > >
> >> > > > > > We initially tried to set it up as a stand-alone utility but
> >> > > eventually
> >> > > > > > gave up. In order to properly do the upgrade, you concurrently
> >> need
> >> > > to
> >> > > > > run
> >> > > > > > whatever upgrade code concurrently with a tablet server
> hosting
> >> > > > !METADATA
> >> > > > > > and a tablet server that can replay WALs. We ended up
> >> duplicating a
> >> > > lot
> >> > > > > of
> >> > > > > > logic already present in master before scrapping that plan. An
> >> > > > > alternative
> >> > > > > > would have been to try to build on MAC, but that was also
> >> > non-trivial
> >> > > > to
> >> > > > > > deploy, so we spliced the code into the existing upgrade path.
> >> How
> >> > do
> >> > > > you
> >> > > > > > feel about that, Drew?
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> >> > drew.farris@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > I'm +1 for a utility that would allow us to go directly from
> >> 1.4
> >> > to
> >> > > > > 1.6.
> >> > > > > > >
> >> > > > > > > In terms of a general policy, I suggest we make this sort of
> >> > > decision
> >> > > > > on
> >> > > > > > a
> >> > > > > > > case by case basis. My unreasonably self-centered intuition
> >> > > suggests
> >> > > > > that
> >> > > > > > > there may be some folks that want to go from 1.4 to 1.6 now
> >> due
> >> > to
> >> > > a
> >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> versions
> >> > like
> >> > > > > might
> >> > > > > > > not exist in the future.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> >> > busbey@cloudera.com>
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > In an effort to get more users off of our now unsupported
> >> 1.4
> >> > > > > release,
> >> > > > > > > > should we support upgrading directly to 1.6 without going
> >> > > through a
> >> > > > > 1.5
> >> > > > > > > > upgrade?
> >> > > > > > > >
> >> > > > > > > > More directly for those on user@: would you be more
> likely
> >> to
> >> > > > > upgrade
> >> > > > > > > off
> >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> >> > > > > > > >
> >> > > > > > > > We have this working locally at Cloudera as a part of our
> >> CDH
> >> > > > > > integration
> >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> >> > > > > > > >
> >> > > > > > > > We can get into implementation details on a jira if
> there's
> >> > > > positive
> >> > > > > > > > consensus, but the changes weren't very complicated.
> They're
> >> > > mostly
> >> > > > > > > >
> >> > > > > > > > * forward porting and consolidating some upgrade code
> >> > > > > > > > * additions to the README for instructions
> >> > > > > > > >
> >> > > > > > > > Personally, I can see the both sides of the argument. On
> the
> >> > plus
> >> > > > > side,
> >> > > > > > > > anything to get more users off of 1.4 is a good thing. On
> >> the
> >> > > > > negative
> >> > > > > > > > side, it means we have the 1.4 related upgrade code
> sitting
> >> in
> >> > a
> >> > > > > > > supported
> >> > > > > > > > code branch longer.
> >> > > > > > > >
> >> > > > > > > > Thoughts?
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Sean
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Sean
> >> >
> >>
> >
> >
> >
> > --
> > Sean
> >
>
>
>
> --
> Sean
>
Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
through 1.5?
Posted by Sean Busbey <bu...@cloudera.com>.
Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
https://reviews.apache.org/r/23413/
Pending any additional concerns, I'll be pushing this soon.
On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com> wrote:
> An initial port of the 1.4 -> 1.6 upgrade code for the current
> 1.6.1-SNAPSHOT branch is now up:
>
> * https://issues.apache.org/jira/browse/ACCUMULO-2988
> * https://reviews.apache.org/r/23413/
>
>
> On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org> wrote:
>
>> Cool. Thanks, Sean!
>>
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>>
>> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>> > I'll be creating a ticket and posting a patch this week.
>> >
>> >
>> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org>
>> wrote:
>> >
>> > > So, just to revisit this conversation, it seems like there is
>> interest in
>> > > supporting this. Is there already a ticket for it and/or somebody
>> > > interested in doing the necessary work for 1.6.1?
>> > >
>> > >
>> > > --
>> > > Christopher L Tubbs II
>> > > http://gravatar.com/ctubbsii
>> > >
>> > >
>> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com>
>> wrote:
>> > >
>> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
>> > > > (ACCUMULO-2770), start 1.6
>> > > >
>> > > > Mike
>> > > >
>> > > >
>> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <drew.farris@gmail.com
>> >
>> > > > wrote:
>> > > >
>> > > > > Mike,
>> > > > >
>> > > > > So works just like upgrading from 1.5?
>> > > > >
>> > > > > (After 1.4 shutdown, install 1.6 and restart?)
>> > > > >
>> > > > > That sounds entirely reasonable.
>> > > > >
>> > > > > Drew
>> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com>
>> wrote:
>> > > > >
>> > > > > > We initially tried to set it up as a stand-alone utility but
>> > > eventually
>> > > > > > gave up. In order to properly do the upgrade, you concurrently
>> need
>> > > to
>> > > > > run
>> > > > > > whatever upgrade code concurrently with a tablet server hosting
>> > > > !METADATA
>> > > > > > and a tablet server that can replay WALs. We ended up
>> duplicating a
>> > > lot
>> > > > > of
>> > > > > > logic already present in master before scrapping that plan. An
>> > > > > alternative
>> > > > > > would have been to try to build on MAC, but that was also
>> > non-trivial
>> > > > to
>> > > > > > deploy, so we spliced the code into the existing upgrade path.
>> How
>> > do
>> > > > you
>> > > > > > feel about that, Drew?
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
>> > drew.farris@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > I'm +1 for a utility that would allow us to go directly from
>> 1.4
>> > to
>> > > > > 1.6.
>> > > > > > >
>> > > > > > > In terms of a general policy, I suggest we make this sort of
>> > > decision
>> > > > > on
>> > > > > > a
>> > > > > > > case by case basis. My unreasonably self-centered intuition
>> > > suggests
>> > > > > that
>> > > > > > > there may be some folks that want to go from 1.4 to 1.6 now
>> due
>> > to
>> > > a
>> > > > > > > relatively short 1.5 cycle. The need to jump multiple versions
>> > like
>> > > > > might
>> > > > > > > not exist in the future.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
>> > busbey@cloudera.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > In an effort to get more users off of our now unsupported
>> 1.4
>> > > > > release,
>> > > > > > > > should we support upgrading directly to 1.6 without going
>> > > through a
>> > > > > 1.5
>> > > > > > > > upgrade?
>> > > > > > > >
>> > > > > > > > More directly for those on user@: would you be more likely
>> to
>> > > > > upgrade
>> > > > > > > off
>> > > > > > > > of 1.4 if you could do so directly to 1.6?
>> > > > > > > >
>> > > > > > > > We have this working locally at Cloudera as a part of our
>> CDH
>> > > > > > integration
>> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
>> > > > > > > >
>> > > > > > > > We can get into implementation details on a jira if there's
>> > > > positive
>> > > > > > > > consensus, but the changes weren't very complicated. They're
>> > > mostly
>> > > > > > > >
>> > > > > > > > * forward porting and consolidating some upgrade code
>> > > > > > > > * additions to the README for instructions
>> > > > > > > >
>> > > > > > > > Personally, I can see the both sides of the argument. On the
>> > plus
>> > > > > side,
>> > > > > > > > anything to get more users off of 1.4 is a good thing. On
>> the
>> > > > > negative
>> > > > > > > > side, it means we have the 1.4 related upgrade code sitting
>> in
>> > a
>> > > > > > > supported
>> > > > > > > > code branch longer.
>> > > > > > > >
>> > > > > > > > Thoughts?
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Sean
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Sean
>> >
>>
>
>
>
> --
> Sean
>
--
Sean