You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Sean Busbey <bu...@cloudera.com> on 2014/06/16 23:24:37 UTC

[DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

In an effort to get more users off of our now unsupported 1.4 release,
should we support upgrading directly to 1.6 without going through a 1.5
upgrade?

More directly for those on user@: would you be more likely to upgrade off
of 1.4 if you could do so directly to 1.6?

We have this working locally at Cloudera as a part of our CDH integration
(we shipped 1.4 and we're planning to ship 1.6 next).

We can get into implementation details on a jira if there's positive
consensus, but the changes weren't very complicated. They're mostly

* forward porting and consolidating some upgrade code
* additions to the README for instructions

Personally, I can see the both sides of the argument. On the plus side,
anything to get more users off of 1.4 is a good thing. On the negative
side, it means we have the 1.4 related upgrade code sitting in a supported
code branch longer.

Thoughts?

-- 
Sean

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Sean Busbey <bu...@cloudera.com>.
On Mon, Jun 16, 2014 at 5:29 PM, John Vines <jv...@gmail.com> wrote:

> My comfort with this is based solely on the amount of effort and
> complexity involved. I have no direct qualms with providing that path, on
> the assumption we don't spend an exorbitant amount of hours developing it
> and end up with a ridiculous amount of code brought up to support this
> which we have to maintain.
>
> I would also be happier to have some sort of story involved for how to
> ensure the 1.4 users migrate to 1.6 in such a way we don't hit a similar
> race condition we found for if/when they jump to 1.7/2.0.
>
>
>
I think the only way to do this is to state that it's a one-off exception.

Presuming we don't want to change our supported upgrade plan to be "upgrade
from  any previous version to current."

-- 
Sean

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Sean Busbey <bu...@cloudera.com>.
On Mon, Jun 16, 2014 at 5:29 PM, John Vines <jv...@gmail.com> wrote:

> My comfort with this is based solely on the amount of effort and
> complexity involved. I have no direct qualms with providing that path, on
> the assumption we don't spend an exorbitant amount of hours developing it
> and end up with a ridiculous amount of code brought up to support this
> which we have to maintain.
>
> I would also be happier to have some sort of story involved for how to
> ensure the 1.4 users migrate to 1.6 in such a way we don't hit a similar
> race condition we found for if/when they jump to 1.7/2.0.
>
>
>
I think the only way to do this is to state that it's a one-off exception.

Presuming we don't want to change our supported upgrade plan to be "upgrade
from  any previous version to current."

-- 
Sean

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by John Vines <jv...@gmail.com>.
My comfort with this is based solely on the amount of effort and complexity
involved. I have no direct qualms with providing that path, on the
assumption we don't spend an exorbitant amount of hours developing it and
end up with a ridiculous amount of code brought up to support this which we
have to maintain.

I would also be happier to have some sort of story involved for how to
ensure the 1.4 users migrate to 1.6 in such a way we don't hit a similar
race condition we found for if/when they jump to 1.7/2.0.


On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:

> In an effort to get more users off of our now unsupported 1.4 release,
> should we support upgrading directly to 1.6 without going through a 1.5
> upgrade?
>
> More directly for those on user@: would you be more likely to upgrade off
> of 1.4 if you could do so directly to 1.6?
>
> We have this working locally at Cloudera as a part of our CDH integration
> (we shipped 1.4 and we're planning to ship 1.6 next).
>
> We can get into implementation details on a jira if there's positive
> consensus, but the changes weren't very complicated. They're mostly
>
> * forward porting and consolidating some upgrade code
> * additions to the README for instructions
>
> Personally, I can see the both sides of the argument. On the plus side,
> anything to get more users off of 1.4 is a good thing. On the negative
> side, it means we have the 1.4 related upgrade code sitting in a supported
> code branch longer.
>
> Thoughts?
>
> --
> Sean
>



-- 
Cheers
~John

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Sean Busbey <bu...@cloudera.com>.
Unfortunately, the way we do updates to the metadata table makes a
standalone utility problematic.

The current implementation relies on the LocalWALRecovery utility to handle
the "move files to HDFS" step and then does the metadata / zk updates in
the master start up the same way we do for other existing upgrade steps.

-Sean


On Mon, Jun 16, 2014 at 5:31 PM, William Slacum <
wilhelm.von.cloud@accumulo.net> wrote:

> How much of this is a standalone utility? I think a magic button approach
> would be good for this case.
>
>
> On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> In an effort to get more users off of our now unsupported 1.4 release,
>> should we support upgrading directly to 1.6 without going through a 1.5
>> upgrade?
>>
>> More directly for those on user@: would you be more likely to upgrade
>> off of 1.4 if you could do so directly to 1.6?
>>
>> We have this working locally at Cloudera as a part of our CDH integration
>> (we shipped 1.4 and we're planning to ship 1.6 next).
>>
>> We can get into implementation details on a jira if there's positive
>> consensus, but the changes weren't very complicated. They're mostly
>>
>> * forward porting and consolidating some upgrade code
>> * additions to the README for instructions
>>
>> Personally, I can see the both sides of the argument. On the plus side,
>> anything to get more users off of 1.4 is a good thing. On the negative
>> side, it means we have the 1.4 related upgrade code sitting in a supported
>> code branch longer.
>>
>> Thoughts?
>>
>> --
>> Sean
>>
>
>


-- 
Sean

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Sean Busbey <bu...@cloudera.com>.
Unfortunately, the way we do updates to the metadata table makes a
standalone utility problematic.

The current implementation relies on the LocalWALRecovery utility to handle
the "move files to HDFS" step and then does the metadata / zk updates in
the master start up the same way we do for other existing upgrade steps.

-Sean


On Mon, Jun 16, 2014 at 5:31 PM, William Slacum <
wilhelm.von.cloud@accumulo.net> wrote:

> How much of this is a standalone utility? I think a magic button approach
> would be good for this case.
>
>
> On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> In an effort to get more users off of our now unsupported 1.4 release,
>> should we support upgrading directly to 1.6 without going through a 1.5
>> upgrade?
>>
>> More directly for those on user@: would you be more likely to upgrade
>> off of 1.4 if you could do so directly to 1.6?
>>
>> We have this working locally at Cloudera as a part of our CDH integration
>> (we shipped 1.4 and we're planning to ship 1.6 next).
>>
>> We can get into implementation details on a jira if there's positive
>> consensus, but the changes weren't very complicated. They're mostly
>>
>> * forward porting and consolidating some upgrade code
>> * additions to the README for instructions
>>
>> Personally, I can see the both sides of the argument. On the plus side,
>> anything to get more users off of 1.4 is a good thing. On the negative
>> side, it means we have the 1.4 related upgrade code sitting in a supported
>> code branch longer.
>>
>> Thoughts?
>>
>> --
>> Sean
>>
>
>


-- 
Sean

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by William Slacum <wi...@accumulo.net>.
How much of this is a standalone utility? I think a magic button approach
would be good for this case.


On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:

> In an effort to get more users off of our now unsupported 1.4 release,
> should we support upgrading directly to 1.6 without going through a 1.5
> upgrade?
>
> More directly for those on user@: would you be more likely to upgrade off
> of 1.4 if you could do so directly to 1.6?
>
> We have this working locally at Cloudera as a part of our CDH integration
> (we shipped 1.4 and we're planning to ship 1.6 next).
>
> We can get into implementation details on a jira if there's positive
> consensus, but the changes weren't very complicated. They're mostly
>
> * forward porting and consolidating some upgrade code
> * additions to the README for instructions
>
> Personally, I can see the both sides of the argument. On the plus side,
> anything to get more users off of 1.4 is a good thing. On the negative
> side, it means we have the 1.4 related upgrade code sitting in a supported
> code branch longer.
>
> Thoughts?
>
> --
> Sean
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Christopher <ct...@apache.org>.
My main concern would be the risk of destabilizing 1.6.x by reintroducing
upgrade code to support upgrading from both 1.4.x and 1.5.x. So long as
that risk is mitigated by making the changeset minimal and thorough
review/testing, I see no issue with providing a direct upgrade path from
1.4.x/1.5.x -> 1.6.1. I wouldn't want to make it a precedent, though. It's
hard enough supporting upgrades from one prior version.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:

> In an effort to get more users off of our now unsupported 1.4 release,
> should we support upgrading directly to 1.6 without going through a 1.5
> upgrade?
>
> More directly for those on user@: would you be more likely to upgrade off
> of 1.4 if you could do so directly to 1.6?
>
> We have this working locally at Cloudera as a part of our CDH integration
> (we shipped 1.4 and we're planning to ship 1.6 next).
>
> We can get into implementation details on a jira if there's positive
> consensus, but the changes weren't very complicated. They're mostly
>
> * forward porting and consolidating some upgrade code
> * additions to the README for instructions
>
> Personally, I can see the both sides of the argument. On the plus side,
> anything to get more users off of 1.4 is a good thing. On the negative
> side, it means we have the 1.4 related upgrade code sitting in a supported
> code branch longer.
>
> Thoughts?
>
> --
> Sean
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by John Vines <jv...@gmail.com>.
My comfort with this is based solely on the amount of effort and complexity
involved. I have no direct qualms with providing that path, on the
assumption we don't spend an exorbitant amount of hours developing it and
end up with a ridiculous amount of code brought up to support this which we
have to maintain.

I would also be happier to have some sort of story involved for how to
ensure the 1.4 users migrate to 1.6 in such a way we don't hit a similar
race condition we found for if/when they jump to 1.7/2.0.


On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:

> In an effort to get more users off of our now unsupported 1.4 release,
> should we support upgrading directly to 1.6 without going through a 1.5
> upgrade?
>
> More directly for those on user@: would you be more likely to upgrade off
> of 1.4 if you could do so directly to 1.6?
>
> We have this working locally at Cloudera as a part of our CDH integration
> (we shipped 1.4 and we're planning to ship 1.6 next).
>
> We can get into implementation details on a jira if there's positive
> consensus, but the changes weren't very complicated. They're mostly
>
> * forward porting and consolidating some upgrade code
> * additions to the README for instructions
>
> Personally, I can see the both sides of the argument. On the plus side,
> anything to get more users off of 1.4 is a good thing. On the negative
> side, it means we have the 1.4 related upgrade code sitting in a supported
> code branch longer.
>
> Thoughts?
>
> --
> Sean
>



-- 
Cheers
~John

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Sean Busbey <bu...@cloudera.com>.
On Wed, Jul 30, 2014 at 5:55 PM, <dl...@comcast.net> wrote:

> Please add a check that instance.dfs.uri, although deprecated, is
> configured in the upgrade and that instance.volumes is NOT configured, see
> the discussion in ACCUMULO-3006.
>
>

Dave, how is that change related to the changes in ACCUMULO-2988?

I'm all for adding that check. However, it's something we should be doing
for all 1.6 upgrades; it isn't a part of adding teh 1.4 -> 1.6 path.

-Sean




> -----Original Message-----
> From: Mike Drob [mailto:madrob@cloudera.com]
> Sent: Wednesday, July 30, 2014 6:12 PM
> To: Accumulo Dev List
> Subject: Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
> through 1.5?
>
> The upgrade test looks something like:
>
> * Start up 1.4 (or 1.5)
> * Create tables with a variety of configurations (LZO/Snappy/GZ, Bloom
> Filter On/Off, Block Cache On/Off)
> * Load some data into these tables, enough such that flushes and
> compactions start to occur.
> * Abruptly kill all of the servers. This ensures that there are WALs
> around. Unfortunately, we can't have any compactions in progress at this
> point, since that causes a Fate Operation, and that would prevent the
> upgrade from completing.
> * Upgrade the bits, start 1.6, and wait for the upgrade steps to finish.
> * Check for data loss
> * Trigger some compactions and wait for those to finish. In a previous
> iteration, we discovered an issue where the relative path'd files and WALs
> were not being properly deleted from !0.
> * Check for data loss.
> * Restart the tablet servers to force tablets to reload (test for orphaned
> files).
> * Check for data loss.
>
> We did not try using custom iterators.
>
> Mike
>
>
> On Wed, Jul 30, 2014 at 4:58 PM, Christopher <ct...@apache.org> wrote:
>
> > Do you have a list of test cases you've tried?
> > (like with/without walogs, custom iterators on tables, delete markers
> > for the metadata table, etc.)?
> >
> > If we could consolidate some of those into a comprehensive list,
> > that'd be pretty useful, I'd think, to give confidence that we didn't
> > miss anything in the review. It might also help focus testing for
> > 1.6.1 release after you've pushed.
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> >
> > On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
> > >
> > > https://reviews.apache.org/r/23413/
> > >
> > > Pending any additional concerns, I'll be pushing this soon.
> > >
> > >
> > > On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > > > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > > > 1.6.1-SNAPSHOT branch is now up:
> > > >
> > > > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > > > * https://reviews.apache.org/r/23413/
> > > >
> > > >
> > > > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> > > wrote:
> > > >
> > > >> Cool. Thanks, Sean!
> > > >>
> > > >>
> > > >> --
> > > >> Christopher L Tubbs II
> > > >> http://gravatar.com/ctubbsii
> > > >>
> > > >>
> > > >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey
> > > >> <bu...@cloudera.com>
> > > wrote:
> > > >>
> > > >> > I'll be creating a ticket and posting a patch this week.
> > > >> >
> > > >> >
> > > >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher
> > > >> > <ct...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > So, just to revisit this conversation, it seems like there is
> > > >> interest in
> > > >> > > supporting this. Is there already a ticket for it and/or
> > > >> > > somebody interested in doing the necessary work for 1.6.1?
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Christopher L Tubbs II
> > > >> > > http://gravatar.com/ctubbsii
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob
> > > >> > > <ma...@cloudera.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > > >> > > > (ACCUMULO-2770), start 1.6
> > > >> > > >
> > > >> > > > Mike
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> > > drew.farris@gmail.com
> > > >> >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Mike,
> > > >> > > > >
> > > >> > > > > So works just like upgrading from 1.5?
> > > >> > > > >
> > > >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > > >> > > > >
> > > >> > > > > That sounds entirely reasonable.
> > > >> > > > >
> > > >> > > > > Drew
> > > >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob"
> > > >> > > > > <ma...@cloudera.com>
> > > >> wrote:
> > > >> > > > >
> > > >> > > > > > We initially tried to set it up as a stand-alone
> > > >> > > > > > utility but
> > > >> > > eventually
> > > >> > > > > > gave up. In order to properly do the upgrade, you
> > concurrently
> > > >> need
> > > >> > > to
> > > >> > > > > run
> > > >> > > > > > whatever upgrade code concurrently with a tablet server
> > > hosting
> > > >> > > > !METADATA
> > > >> > > > > > and a tablet server that can replay WALs. We ended up
> > > >> duplicating a
> > > >> > > lot
> > > >> > > > > of
> > > >> > > > > > logic already present in master before scrapping that
> plan.
> > An
> > > >> > > > > alternative
> > > >> > > > > > would have been to try to build on MAC, but that was
> > > >> > > > > > also
> > > >> > non-trivial
> > > >> > > > to
> > > >> > > > > > deploy, so we spliced the code into the existing
> > > >> > > > > > upgrade
> > path.
> > > >> How
> > > >> > do
> > > >> > > > you
> > > >> > > > > > feel about that, Drew?
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > > >> > drew.farris@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > I'm +1 for a utility that would allow us to go
> > > >> > > > > > > directly
> > from
> > > >> 1.4
> > > >> > to
> > > >> > > > > 1.6.
> > > >> > > > > > >
> > > >> > > > > > > In terms of a general policy, I suggest we make this
> > > >> > > > > > > sort
> > of
> > > >> > > decision
> > > >> > > > > on
> > > >> > > > > > a
> > > >> > > > > > > case by case basis. My unreasonably self-centered
> > intuition
> > > >> > > suggests
> > > >> > > > > that
> > > >> > > > > > > there may be some folks that want to go from 1.4 to
> > > >> > > > > > > 1.6
> > now
> > > >> due
> > > >> > to
> > > >> > > a
> > > >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> > > versions
> > > >> > like
> > > >> > > > > might
> > > >> > > > > > > not exist in the future.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > > >> > busbey@cloudera.com>
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > In an effort to get more users off of our now
> > unsupported
> > > >> 1.4
> > > >> > > > > release,
> > > >> > > > > > > > should we support upgrading directly to 1.6 without
> > going
> > > >> > > through a
> > > >> > > > > 1.5
> > > >> > > > > > > > upgrade?
> > > >> > > > > > > >
> > > >> > > > > > > > More directly for those on user@: would you be more
> > > likely
> > > >> to
> > > >> > > > > upgrade
> > > >> > > > > > > off
> > > >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > > >> > > > > > > >
> > > >> > > > > > > > We have this working locally at Cloudera as a part
> > > >> > > > > > > > of
> > our
> > > >> CDH
> > > >> > > > > > integration
> > > >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > >> > > > > > > >
> > > >> > > > > > > > We can get into implementation details on a jira if
> > > there's
> > > >> > > > positive
> > > >> > > > > > > > consensus, but the changes weren't very complicated.
> > > They're
> > > >> > > mostly
> > > >> > > > > > > >
> > > >> > > > > > > > * forward porting and consolidating some upgrade
> > > >> > > > > > > > code
> > > >> > > > > > > > * additions to the README for instructions
> > > >> > > > > > > >
> > > >> > > > > > > > Personally, I can see the both sides of the
> > > >> > > > > > > > argument. On
> > > the
> > > >> > plus
> > > >> > > > > side,
> > > >> > > > > > > > anything to get more users off of 1.4 is a good thing.
> > On
> > > >> the
> > > >> > > > > negative
> > > >> > > > > > > > side, it means we have the 1.4 related upgrade code
> > > sitting
> > > >> in
> > > >> > a
> > > >> > > > > > > supported
> > > >> > > > > > > > code branch longer.
> > > >> > > > > > > >
> > > >> > > > > > > > Thoughts?
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > > Sean
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Sean
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
>
>


-- 
Sean

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Mike Drob <ma...@cloudera.com>.
That seems out of scope for this issue. Can it be done as part of the work
on ACCUMULO-3006 when that gets in?


On Wed, Jul 30, 2014 at 5:55 PM, <dl...@comcast.net> wrote:

> Please add a check that instance.dfs.uri, although deprecated, is
> configured in the upgrade and that instance.volumes is NOT configured, see
> the discussion in ACCUMULO-3006.
>
> -----Original Message-----
> From: Mike Drob [mailto:madrob@cloudera.com]
> Sent: Wednesday, July 30, 2014 6:12 PM
> To: Accumulo Dev List
> Subject: Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going
> through 1.5?
>
> The upgrade test looks something like:
>
> * Start up 1.4 (or 1.5)
> * Create tables with a variety of configurations (LZO/Snappy/GZ, Bloom
> Filter On/Off, Block Cache On/Off)
> * Load some data into these tables, enough such that flushes and
> compactions start to occur.
> * Abruptly kill all of the servers. This ensures that there are WALs
> around. Unfortunately, we can't have any compactions in progress at this
> point, since that causes a Fate Operation, and that would prevent the
> upgrade from completing.
> * Upgrade the bits, start 1.6, and wait for the upgrade steps to finish.
> * Check for data loss
> * Trigger some compactions and wait for those to finish. In a previous
> iteration, we discovered an issue where the relative path'd files and WALs
> were not being properly deleted from !0.
> * Check for data loss.
> * Restart the tablet servers to force tablets to reload (test for orphaned
> files).
> * Check for data loss.
>
> We did not try using custom iterators.
>
> Mike
>
>
> On Wed, Jul 30, 2014 at 4:58 PM, Christopher <ct...@apache.org> wrote:
>
> > Do you have a list of test cases you've tried?
> > (like with/without walogs, custom iterators on tables, delete markers
> > for the metadata table, etc.)?
> >
> > If we could consolidate some of those into a comprehensive list,
> > that'd be pretty useful, I'd think, to give confidence that we didn't
> > miss anything in the review. It might also help focus testing for
> > 1.6.1 release after you've pushed.
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> >
> > On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
> > >
> > > https://reviews.apache.org/r/23413/
> > >
> > > Pending any additional concerns, I'll be pushing this soon.
> > >
> > >
> > > On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > > > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > > > 1.6.1-SNAPSHOT branch is now up:
> > > >
> > > > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > > > * https://reviews.apache.org/r/23413/
> > > >
> > > >
> > > > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> > > wrote:
> > > >
> > > >> Cool. Thanks, Sean!
> > > >>
> > > >>
> > > >> --
> > > >> Christopher L Tubbs II
> > > >> http://gravatar.com/ctubbsii
> > > >>
> > > >>
> > > >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey
> > > >> <bu...@cloudera.com>
> > > wrote:
> > > >>
> > > >> > I'll be creating a ticket and posting a patch this week.
> > > >> >
> > > >> >
> > > >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher
> > > >> > <ct...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > So, just to revisit this conversation, it seems like there is
> > > >> interest in
> > > >> > > supporting this. Is there already a ticket for it and/or
> > > >> > > somebody interested in doing the necessary work for 1.6.1?
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Christopher L Tubbs II
> > > >> > > http://gravatar.com/ctubbsii
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob
> > > >> > > <ma...@cloudera.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > > >> > > > (ACCUMULO-2770), start 1.6
> > > >> > > >
> > > >> > > > Mike
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> > > drew.farris@gmail.com
> > > >> >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Mike,
> > > >> > > > >
> > > >> > > > > So works just like upgrading from 1.5?
> > > >> > > > >
> > > >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > > >> > > > >
> > > >> > > > > That sounds entirely reasonable.
> > > >> > > > >
> > > >> > > > > Drew
> > > >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob"
> > > >> > > > > <ma...@cloudera.com>
> > > >> wrote:
> > > >> > > > >
> > > >> > > > > > We initially tried to set it up as a stand-alone
> > > >> > > > > > utility but
> > > >> > > eventually
> > > >> > > > > > gave up. In order to properly do the upgrade, you
> > concurrently
> > > >> need
> > > >> > > to
> > > >> > > > > run
> > > >> > > > > > whatever upgrade code concurrently with a tablet server
> > > hosting
> > > >> > > > !METADATA
> > > >> > > > > > and a tablet server that can replay WALs. We ended up
> > > >> duplicating a
> > > >> > > lot
> > > >> > > > > of
> > > >> > > > > > logic already present in master before scrapping that
> plan.
> > An
> > > >> > > > > alternative
> > > >> > > > > > would have been to try to build on MAC, but that was
> > > >> > > > > > also
> > > >> > non-trivial
> > > >> > > > to
> > > >> > > > > > deploy, so we spliced the code into the existing
> > > >> > > > > > upgrade
> > path.
> > > >> How
> > > >> > do
> > > >> > > > you
> > > >> > > > > > feel about that, Drew?
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > > >> > drew.farris@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > I'm +1 for a utility that would allow us to go
> > > >> > > > > > > directly
> > from
> > > >> 1.4
> > > >> > to
> > > >> > > > > 1.6.
> > > >> > > > > > >
> > > >> > > > > > > In terms of a general policy, I suggest we make this
> > > >> > > > > > > sort
> > of
> > > >> > > decision
> > > >> > > > > on
> > > >> > > > > > a
> > > >> > > > > > > case by case basis. My unreasonably self-centered
> > intuition
> > > >> > > suggests
> > > >> > > > > that
> > > >> > > > > > > there may be some folks that want to go from 1.4 to
> > > >> > > > > > > 1.6
> > now
> > > >> due
> > > >> > to
> > > >> > > a
> > > >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> > > versions
> > > >> > like
> > > >> > > > > might
> > > >> > > > > > > not exist in the future.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > > >> > busbey@cloudera.com>
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > In an effort to get more users off of our now
> > unsupported
> > > >> 1.4
> > > >> > > > > release,
> > > >> > > > > > > > should we support upgrading directly to 1.6 without
> > going
> > > >> > > through a
> > > >> > > > > 1.5
> > > >> > > > > > > > upgrade?
> > > >> > > > > > > >
> > > >> > > > > > > > More directly for those on user@: would you be more
> > > likely
> > > >> to
> > > >> > > > > upgrade
> > > >> > > > > > > off
> > > >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > > >> > > > > > > >
> > > >> > > > > > > > We have this working locally at Cloudera as a part
> > > >> > > > > > > > of
> > our
> > > >> CDH
> > > >> > > > > > integration
> > > >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > >> > > > > > > >
> > > >> > > > > > > > We can get into implementation details on a jira if
> > > there's
> > > >> > > > positive
> > > >> > > > > > > > consensus, but the changes weren't very complicated.
> > > They're
> > > >> > > mostly
> > > >> > > > > > > >
> > > >> > > > > > > > * forward porting and consolidating some upgrade
> > > >> > > > > > > > code
> > > >> > > > > > > > * additions to the README for instructions
> > > >> > > > > > > >
> > > >> > > > > > > > Personally, I can see the both sides of the
> > > >> > > > > > > > argument. On
> > > the
> > > >> > plus
> > > >> > > > > side,
> > > >> > > > > > > > anything to get more users off of 1.4 is a good thing.
> > On
> > > >> the
> > > >> > > > > negative
> > > >> > > > > > > > side, it means we have the 1.4 related upgrade code
> > > sitting
> > > >> in
> > > >> > a
> > > >> > > > > > > supported
> > > >> > > > > > > > code branch longer.
> > > >> > > > > > > >
> > > >> > > > > > > > Thoughts?
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > > Sean
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Sean
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
>
>

RE: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by dl...@comcast.net.
Please add a check that instance.dfs.uri, although deprecated, is configured in the upgrade and that instance.volumes is NOT configured, see the discussion in ACCUMULO-3006.

-----Original Message-----
From: Mike Drob [mailto:madrob@cloudera.com] 
Sent: Wednesday, July 30, 2014 6:12 PM
To: Accumulo Dev List
Subject: Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

The upgrade test looks something like:

* Start up 1.4 (or 1.5)
* Create tables with a variety of configurations (LZO/Snappy/GZ, Bloom Filter On/Off, Block Cache On/Off)
* Load some data into these tables, enough such that flushes and compactions start to occur.
* Abruptly kill all of the servers. This ensures that there are WALs around. Unfortunately, we can't have any compactions in progress at this point, since that causes a Fate Operation, and that would prevent the upgrade from completing.
* Upgrade the bits, start 1.6, and wait for the upgrade steps to finish.
* Check for data loss
* Trigger some compactions and wait for those to finish. In a previous iteration, we discovered an issue where the relative path'd files and WALs were not being properly deleted from !0.
* Check for data loss.
* Restart the tablet servers to force tablets to reload (test for orphaned files).
* Check for data loss.

We did not try using custom iterators.

Mike


On Wed, Jul 30, 2014 at 4:58 PM, Christopher <ct...@apache.org> wrote:

> Do you have a list of test cases you've tried?
> (like with/without walogs, custom iterators on tables, delete markers 
> for the metadata table, etc.)?
>
> If we could consolidate some of those into a comprehensive list, 
> that'd be pretty useful, I'd think, to give confidence that we didn't 
> miss anything in the review. It might also help focus testing for 
> 1.6.1 release after you've pushed.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
> >
> > https://reviews.apache.org/r/23413/
> >
> > Pending any additional concerns, I'll be pushing this soon.
> >
> >
> > On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > An initial port of the 1.4 -> 1.6 upgrade code for the current 
> > > 1.6.1-SNAPSHOT branch is now up:
> > >
> > > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > > * https://reviews.apache.org/r/23413/
> > >
> > >
> > > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> > wrote:
> > >
> > >> Cool. Thanks, Sean!
> > >>
> > >>
> > >> --
> > >> Christopher L Tubbs II
> > >> http://gravatar.com/ctubbsii
> > >>
> > >>
> > >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey 
> > >> <bu...@cloudera.com>
> > wrote:
> > >>
> > >> > I'll be creating a ticket and posting a patch this week.
> > >> >
> > >> >
> > >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher 
> > >> > <ct...@apache.org>
> > >> wrote:
> > >> >
> > >> > > So, just to revisit this conversation, it seems like there is
> > >> interest in
> > >> > > supporting this. Is there already a ticket for it and/or 
> > >> > > somebody interested in doing the necessary work for 1.6.1?
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Christopher L Tubbs II
> > >> > > http://gravatar.com/ctubbsii
> > >> > >
> > >> > >
> > >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob 
> > >> > > <ma...@cloudera.com>
> > >> wrote:
> > >> > >
> > >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS 
> > >> > > > (ACCUMULO-2770), start 1.6
> > >> > > >
> > >> > > > Mike
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> > drew.farris@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Mike,
> > >> > > > >
> > >> > > > > So works just like upgrading from 1.5?
> > >> > > > >
> > >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > >> > > > >
> > >> > > > > That sounds entirely reasonable.
> > >> > > > >
> > >> > > > > Drew
> > >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob" 
> > >> > > > > <ma...@cloudera.com>
> > >> wrote:
> > >> > > > >
> > >> > > > > > We initially tried to set it up as a stand-alone 
> > >> > > > > > utility but
> > >> > > eventually
> > >> > > > > > gave up. In order to properly do the upgrade, you
> concurrently
> > >> need
> > >> > > to
> > >> > > > > run
> > >> > > > > > whatever upgrade code concurrently with a tablet server
> > hosting
> > >> > > > !METADATA
> > >> > > > > > and a tablet server that can replay WALs. We ended up
> > >> duplicating a
> > >> > > lot
> > >> > > > > of
> > >> > > > > > logic already present in master before scrapping that plan.
> An
> > >> > > > > alternative
> > >> > > > > > would have been to try to build on MAC, but that was 
> > >> > > > > > also
> > >> > non-trivial
> > >> > > > to
> > >> > > > > > deploy, so we spliced the code into the existing 
> > >> > > > > > upgrade
> path.
> > >> How
> > >> > do
> > >> > > > you
> > >> > > > > > feel about that, Drew?
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > >> > drew.farris@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > I'm +1 for a utility that would allow us to go 
> > >> > > > > > > directly
> from
> > >> 1.4
> > >> > to
> > >> > > > > 1.6.
> > >> > > > > > >
> > >> > > > > > > In terms of a general policy, I suggest we make this 
> > >> > > > > > > sort
> of
> > >> > > decision
> > >> > > > > on
> > >> > > > > > a
> > >> > > > > > > case by case basis. My unreasonably self-centered
> intuition
> > >> > > suggests
> > >> > > > > that
> > >> > > > > > > there may be some folks that want to go from 1.4 to 
> > >> > > > > > > 1.6
> now
> > >> due
> > >> > to
> > >> > > a
> > >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> > versions
> > >> > like
> > >> > > > > might
> > >> > > > > > > not exist in the future.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > >> > busbey@cloudera.com>
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > In an effort to get more users off of our now
> unsupported
> > >> 1.4
> > >> > > > > release,
> > >> > > > > > > > should we support upgrading directly to 1.6 without
> going
> > >> > > through a
> > >> > > > > 1.5
> > >> > > > > > > > upgrade?
> > >> > > > > > > >
> > >> > > > > > > > More directly for those on user@: would you be more
> > likely
> > >> to
> > >> > > > > upgrade
> > >> > > > > > > off
> > >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > >> > > > > > > >
> > >> > > > > > > > We have this working locally at Cloudera as a part 
> > >> > > > > > > > of
> our
> > >> CDH
> > >> > > > > > integration
> > >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > >> > > > > > > >
> > >> > > > > > > > We can get into implementation details on a jira if
> > there's
> > >> > > > positive
> > >> > > > > > > > consensus, but the changes weren't very complicated.
> > They're
> > >> > > mostly
> > >> > > > > > > >
> > >> > > > > > > > * forward porting and consolidating some upgrade 
> > >> > > > > > > > code
> > >> > > > > > > > * additions to the README for instructions
> > >> > > > > > > >
> > >> > > > > > > > Personally, I can see the both sides of the 
> > >> > > > > > > > argument. On
> > the
> > >> > plus
> > >> > > > > side,
> > >> > > > > > > > anything to get more users off of 1.4 is a good thing.
> On
> > >> the
> > >> > > > > negative
> > >> > > > > > > > side, it means we have the 1.4 related upgrade code
> > sitting
> > >> in
> > >> > a
> > >> > > > > > > supported
> > >> > > > > > > > code branch longer.
> > >> > > > > > > >
> > >> > > > > > > > Thoughts?
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > > Sean
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Sean
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
> >
> >
> > --
> > Sean
> >
>


Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Mike Drob <ma...@cloudera.com>.
The upgrade test looks something like:

* Start up 1.4 (or 1.5)
* Create tables with a variety of configurations (LZO/Snappy/GZ, Bloom
Filter On/Off, Block Cache On/Off)
* Load some data into these tables, enough such that flushes and
compactions start to occur.
* Abruptly kill all of the servers. This ensures that there are WALs
around. Unfortunately, we can't have any compactions in progress at this
point, since that causes a Fate Operation, and that would prevent the
upgrade from completing.
* Upgrade the bits, start 1.6, and wait for the upgrade steps to finish.
* Check for data loss
* Trigger some compactions and wait for those to finish. In a previous
iteration, we discovered an issue where the relative path'd files and WALs
were not being properly deleted from !0.
* Check for data loss.
* Restart the tablet servers to force tablets to reload (test for orphaned
files).
* Check for data loss.

We did not try using custom iterators.

Mike


On Wed, Jul 30, 2014 at 4:58 PM, Christopher <ct...@apache.org> wrote:

> Do you have a list of test cases you've tried?
> (like with/without walogs, custom iterators on tables, delete markers for
> the metadata table, etc.)?
>
> If we could consolidate some of those into a comprehensive list, that'd be
> pretty useful, I'd think, to give confidence that we didn't miss anything
> in the review. It might also help focus testing for 1.6.1 release after
> you've pushed.
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
> >
> > https://reviews.apache.org/r/23413/
> >
> > Pending any additional concerns, I'll be pushing this soon.
> >
> >
> > On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > > 1.6.1-SNAPSHOT branch is now up:
> > >
> > > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > > * https://reviews.apache.org/r/23413/
> > >
> > >
> > > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> > wrote:
> > >
> > >> Cool. Thanks, Sean!
> > >>
> > >>
> > >> --
> > >> Christopher L Tubbs II
> > >> http://gravatar.com/ctubbsii
> > >>
> > >>
> > >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >>
> > >> > I'll be creating a ticket and posting a patch this week.
> > >> >
> > >> >
> > >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org>
> > >> wrote:
> > >> >
> > >> > > So, just to revisit this conversation, it seems like there is
> > >> interest in
> > >> > > supporting this. Is there already a ticket for it and/or somebody
> > >> > > interested in doing the necessary work for 1.6.1?
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Christopher L Tubbs II
> > >> > > http://gravatar.com/ctubbsii
> > >> > >
> > >> > >
> > >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com>
> > >> wrote:
> > >> > >
> > >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > >> > > > (ACCUMULO-2770), start 1.6
> > >> > > >
> > >> > > > Mike
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> > drew.farris@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Mike,
> > >> > > > >
> > >> > > > > So works just like upgrading from 1.5?
> > >> > > > >
> > >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > >> > > > >
> > >> > > > > That sounds entirely reasonable.
> > >> > > > >
> > >> > > > > Drew
> > >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com>
> > >> wrote:
> > >> > > > >
> > >> > > > > > We initially tried to set it up as a stand-alone utility but
> > >> > > eventually
> > >> > > > > > gave up. In order to properly do the upgrade, you
> concurrently
> > >> need
> > >> > > to
> > >> > > > > run
> > >> > > > > > whatever upgrade code concurrently with a tablet server
> > hosting
> > >> > > > !METADATA
> > >> > > > > > and a tablet server that can replay WALs. We ended up
> > >> duplicating a
> > >> > > lot
> > >> > > > > of
> > >> > > > > > logic already present in master before scrapping that plan.
> An
> > >> > > > > alternative
> > >> > > > > > would have been to try to build on MAC, but that was also
> > >> > non-trivial
> > >> > > > to
> > >> > > > > > deploy, so we spliced the code into the existing upgrade
> path.
> > >> How
> > >> > do
> > >> > > > you
> > >> > > > > > feel about that, Drew?
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > >> > drew.farris@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > I'm +1 for a utility that would allow us to go directly
> from
> > >> 1.4
> > >> > to
> > >> > > > > 1.6.
> > >> > > > > > >
> > >> > > > > > > In terms of a general policy, I suggest we make this sort
> of
> > >> > > decision
> > >> > > > > on
> > >> > > > > > a
> > >> > > > > > > case by case basis. My unreasonably self-centered
> intuition
> > >> > > suggests
> > >> > > > > that
> > >> > > > > > > there may be some folks that want to go from 1.4 to 1.6
> now
> > >> due
> > >> > to
> > >> > > a
> > >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> > versions
> > >> > like
> > >> > > > > might
> > >> > > > > > > not exist in the future.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > >> > busbey@cloudera.com>
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > In an effort to get more users off of our now
> unsupported
> > >> 1.4
> > >> > > > > release,
> > >> > > > > > > > should we support upgrading directly to 1.6 without
> going
> > >> > > through a
> > >> > > > > 1.5
> > >> > > > > > > > upgrade?
> > >> > > > > > > >
> > >> > > > > > > > More directly for those on user@: would you be more
> > likely
> > >> to
> > >> > > > > upgrade
> > >> > > > > > > off
> > >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > >> > > > > > > >
> > >> > > > > > > > We have this working locally at Cloudera as a part of
> our
> > >> CDH
> > >> > > > > > integration
> > >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > >> > > > > > > >
> > >> > > > > > > > We can get into implementation details on a jira if
> > there's
> > >> > > > positive
> > >> > > > > > > > consensus, but the changes weren't very complicated.
> > They're
> > >> > > mostly
> > >> > > > > > > >
> > >> > > > > > > > * forward porting and consolidating some upgrade code
> > >> > > > > > > > * additions to the README for instructions
> > >> > > > > > > >
> > >> > > > > > > > Personally, I can see the both sides of the argument. On
> > the
> > >> > plus
> > >> > > > > side,
> > >> > > > > > > > anything to get more users off of 1.4 is a good thing.
> On
> > >> the
> > >> > > > > negative
> > >> > > > > > > > side, it means we have the 1.4 related upgrade code
> > sitting
> > >> in
> > >> > a
> > >> > > > > > > supported
> > >> > > > > > > > code branch longer.
> > >> > > > > > > >
> > >> > > > > > > > Thoughts?
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > > Sean
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Sean
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
> >
> >
> > --
> > Sean
> >
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Christopher <ct...@apache.org>.
Do you have a list of test cases you've tried?
(like with/without walogs, custom iterators on tables, delete markers for
the metadata table, etc.)?

If we could consolidate some of those into a comprehensive list, that'd be
pretty useful, I'd think, to give confidence that we didn't miss anything
in the review. It might also help focus testing for 1.6.1 release after
you've pushed.


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Wed, Jul 30, 2014 at 4:39 PM, Sean Busbey <bu...@cloudera.com> wrote:

> Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:
>
> https://reviews.apache.org/r/23413/
>
> Pending any additional concerns, I'll be pushing this soon.
>
>
> On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > An initial port of the 1.4 -> 1.6 upgrade code for the current
> > 1.6.1-SNAPSHOT branch is now up:
> >
> > * https://issues.apache.org/jira/browse/ACCUMULO-2988
> > * https://reviews.apache.org/r/23413/
> >
> >
> > On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org>
> wrote:
> >
> >> Cool. Thanks, Sean!
> >>
> >>
> >> --
> >> Christopher L Tubbs II
> >> http://gravatar.com/ctubbsii
> >>
> >>
> >> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >>
> >> > I'll be creating a ticket and posting a patch this week.
> >> >
> >> >
> >> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org>
> >> wrote:
> >> >
> >> > > So, just to revisit this conversation, it seems like there is
> >> interest in
> >> > > supporting this. Is there already a ticket for it and/or somebody
> >> > > interested in doing the necessary work for 1.6.1?
> >> > >
> >> > >
> >> > > --
> >> > > Christopher L Tubbs II
> >> > > http://gravatar.com/ctubbsii
> >> > >
> >> > >
> >> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com>
> >> wrote:
> >> > >
> >> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> >> > > > (ACCUMULO-2770), start 1.6
> >> > > >
> >> > > > Mike
> >> > > >
> >> > > >
> >> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <
> drew.farris@gmail.com
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > Mike,
> >> > > > >
> >> > > > > So works just like upgrading from 1.5?
> >> > > > >
> >> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> >> > > > >
> >> > > > > That sounds entirely reasonable.
> >> > > > >
> >> > > > > Drew
> >> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com>
> >> wrote:
> >> > > > >
> >> > > > > > We initially tried to set it up as a stand-alone utility but
> >> > > eventually
> >> > > > > > gave up. In order to properly do the upgrade, you concurrently
> >> need
> >> > > to
> >> > > > > run
> >> > > > > > whatever upgrade code concurrently with a tablet server
> hosting
> >> > > > !METADATA
> >> > > > > > and a tablet server that can replay WALs. We ended up
> >> duplicating a
> >> > > lot
> >> > > > > of
> >> > > > > > logic already present in master before scrapping that plan. An
> >> > > > > alternative
> >> > > > > > would have been to try to build on MAC, but that was also
> >> > non-trivial
> >> > > > to
> >> > > > > > deploy, so we spliced the code into the existing upgrade path.
> >> How
> >> > do
> >> > > > you
> >> > > > > > feel about that, Drew?
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> >> > drew.farris@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > I'm +1 for a utility that would allow us to go directly from
> >> 1.4
> >> > to
> >> > > > > 1.6.
> >> > > > > > >
> >> > > > > > > In terms of a general policy, I suggest we make this sort of
> >> > > decision
> >> > > > > on
> >> > > > > > a
> >> > > > > > > case by case basis. My unreasonably self-centered intuition
> >> > > suggests
> >> > > > > that
> >> > > > > > > there may be some folks that want to go from 1.4 to 1.6 now
> >> due
> >> > to
> >> > > a
> >> > > > > > > relatively short 1.5 cycle. The need to jump multiple
> versions
> >> > like
> >> > > > > might
> >> > > > > > > not exist in the future.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> >> > busbey@cloudera.com>
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > In an effort to get more users off of our now unsupported
> >> 1.4
> >> > > > > release,
> >> > > > > > > > should we support upgrading directly to 1.6 without going
> >> > > through a
> >> > > > > 1.5
> >> > > > > > > > upgrade?
> >> > > > > > > >
> >> > > > > > > > More directly for those on user@: would you be more
> likely
> >> to
> >> > > > > upgrade
> >> > > > > > > off
> >> > > > > > > > of 1.4 if you could do so directly to 1.6?
> >> > > > > > > >
> >> > > > > > > > We have this working locally at Cloudera as a part of our
> >> CDH
> >> > > > > > integration
> >> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> >> > > > > > > >
> >> > > > > > > > We can get into implementation details on a jira if
> there's
> >> > > > positive
> >> > > > > > > > consensus, but the changes weren't very complicated.
> They're
> >> > > mostly
> >> > > > > > > >
> >> > > > > > > > * forward porting and consolidating some upgrade code
> >> > > > > > > > * additions to the README for instructions
> >> > > > > > > >
> >> > > > > > > > Personally, I can see the both sides of the argument. On
> the
> >> > plus
> >> > > > > side,
> >> > > > > > > > anything to get more users off of 1.4 is a good thing. On
> >> the
> >> > > > > negative
> >> > > > > > > > side, it means we have the 1.4 related upgrade code
> sitting
> >> in
> >> > a
> >> > > > > > > supported
> >> > > > > > > > code branch longer.
> >> > > > > > > >
> >> > > > > > > > Thoughts?
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Sean
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Sean
> >> >
> >>
> >
> >
> >
> > --
> > Sean
> >
>
>
>
> --
> Sean
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Sean Busbey <bu...@cloudera.com>.
Friendly reminder that direct upgrades from 1.4 to 1.6 is under review:

https://reviews.apache.org/r/23413/

Pending any additional concerns, I'll be pushing this soon.


On Fri, Jul 11, 2014 at 2:22 PM, Sean Busbey <bu...@cloudera.com> wrote:

> An initial port of the 1.4 -> 1.6 upgrade code for the current
> 1.6.1-SNAPSHOT branch is now up:
>
> * https://issues.apache.org/jira/browse/ACCUMULO-2988
> * https://reviews.apache.org/r/23413/
>
>
> On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org> wrote:
>
>> Cool. Thanks, Sean!
>>
>>
>> --
>> Christopher L Tubbs II
>> http://gravatar.com/ctubbsii
>>
>>
>> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>> > I'll be creating a ticket and posting a patch this week.
>> >
>> >
>> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org>
>> wrote:
>> >
>> > > So, just to revisit this conversation, it seems like there is
>> interest in
>> > > supporting this. Is there already a ticket for it and/or somebody
>> > > interested in doing the necessary work for 1.6.1?
>> > >
>> > >
>> > > --
>> > > Christopher L Tubbs II
>> > > http://gravatar.com/ctubbsii
>> > >
>> > >
>> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com>
>> wrote:
>> > >
>> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
>> > > > (ACCUMULO-2770), start 1.6
>> > > >
>> > > > Mike
>> > > >
>> > > >
>> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <drew.farris@gmail.com
>> >
>> > > > wrote:
>> > > >
>> > > > > Mike,
>> > > > >
>> > > > > So works just like upgrading from 1.5?
>> > > > >
>> > > > > (After 1.4 shutdown, install 1.6 and restart?)
>> > > > >
>> > > > > That sounds entirely reasonable.
>> > > > >
>> > > > > Drew
>> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com>
>> wrote:
>> > > > >
>> > > > > > We initially tried to set it up as a stand-alone utility but
>> > > eventually
>> > > > > > gave up. In order to properly do the upgrade, you concurrently
>> need
>> > > to
>> > > > > run
>> > > > > > whatever upgrade code concurrently with a tablet server hosting
>> > > > !METADATA
>> > > > > > and a tablet server that can replay WALs. We ended up
>> duplicating a
>> > > lot
>> > > > > of
>> > > > > > logic already present in master before scrapping that plan. An
>> > > > > alternative
>> > > > > > would have been to try to build on MAC, but that was also
>> > non-trivial
>> > > > to
>> > > > > > deploy, so we spliced the code into the existing upgrade path.
>> How
>> > do
>> > > > you
>> > > > > > feel about that, Drew?
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
>> > drew.farris@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > I'm +1 for a utility that would allow us to go directly from
>> 1.4
>> > to
>> > > > > 1.6.
>> > > > > > >
>> > > > > > > In terms of a general policy, I suggest we make this sort of
>> > > decision
>> > > > > on
>> > > > > > a
>> > > > > > > case by case basis. My unreasonably self-centered intuition
>> > > suggests
>> > > > > that
>> > > > > > > there may be some folks that want to go from 1.4 to 1.6 now
>> due
>> > to
>> > > a
>> > > > > > > relatively short 1.5 cycle. The need to jump multiple versions
>> > like
>> > > > > might
>> > > > > > > not exist in the future.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
>> > busbey@cloudera.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > In an effort to get more users off of our now unsupported
>> 1.4
>> > > > > release,
>> > > > > > > > should we support upgrading directly to 1.6 without going
>> > > through a
>> > > > > 1.5
>> > > > > > > > upgrade?
>> > > > > > > >
>> > > > > > > > More directly for those on user@: would you be more likely
>> to
>> > > > > upgrade
>> > > > > > > off
>> > > > > > > > of 1.4 if you could do so directly to 1.6?
>> > > > > > > >
>> > > > > > > > We have this working locally at Cloudera as a part of our
>> CDH
>> > > > > > integration
>> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
>> > > > > > > >
>> > > > > > > > We can get into implementation details on a jira if there's
>> > > > positive
>> > > > > > > > consensus, but the changes weren't very complicated. They're
>> > > mostly
>> > > > > > > >
>> > > > > > > > * forward porting and consolidating some upgrade code
>> > > > > > > > * additions to the README for instructions
>> > > > > > > >
>> > > > > > > > Personally, I can see the both sides of the argument. On the
>> > plus
>> > > > > side,
>> > > > > > > > anything to get more users off of 1.4 is a good thing. On
>> the
>> > > > > negative
>> > > > > > > > side, it means we have the 1.4 related upgrade code sitting
>> in
>> > a
>> > > > > > > supported
>> > > > > > > > code branch longer.
>> > > > > > > >
>> > > > > > > > Thoughts?
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Sean
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Sean
>> >
>>
>
>
>
> --
> Sean
>



-- 
Sean

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Sean Busbey <bu...@cloudera.com>.
An initial port of the 1.4 -> 1.6 upgrade code for the current
1.6.1-SNAPSHOT branch is now up:

* https://issues.apache.org/jira/browse/ACCUMULO-2988
* https://reviews.apache.org/r/23413/


On Wed, Jun 25, 2014 at 2:36 PM, Christopher <ct...@apache.org> wrote:

> Cool. Thanks, Sean!
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > I'll be creating a ticket and posting a patch this week.
> >
> >
> > On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org>
> wrote:
> >
> > > So, just to revisit this conversation, it seems like there is interest
> in
> > > supporting this. Is there already a ticket for it and/or somebody
> > > interested in doing the necessary work for 1.6.1?
> > >
> > >
> > > --
> > > Christopher L Tubbs II
> > > http://gravatar.com/ctubbsii
> > >
> > >
> > > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com>
> wrote:
> > >
> > > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > > > (ACCUMULO-2770), start 1.6
> > > >
> > > > Mike
> > > >
> > > >
> > > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <dr...@gmail.com>
> > > > wrote:
> > > >
> > > > > Mike,
> > > > >
> > > > > So works just like upgrading from 1.5?
> > > > >
> > > > > (After 1.4 shutdown, install 1.6 and restart?)
> > > > >
> > > > > That sounds entirely reasonable.
> > > > >
> > > > > Drew
> > > > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com> wrote:
> > > > >
> > > > > > We initially tried to set it up as a stand-alone utility but
> > > eventually
> > > > > > gave up. In order to properly do the upgrade, you concurrently
> need
> > > to
> > > > > run
> > > > > > whatever upgrade code concurrently with a tablet server hosting
> > > > !METADATA
> > > > > > and a tablet server that can replay WALs. We ended up
> duplicating a
> > > lot
> > > > > of
> > > > > > logic already present in master before scrapping that plan. An
> > > > > alternative
> > > > > > would have been to try to build on MAC, but that was also
> > non-trivial
> > > > to
> > > > > > deploy, so we spliced the code into the existing upgrade path.
> How
> > do
> > > > you
> > > > > > feel about that, Drew?
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> > drew.farris@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I'm +1 for a utility that would allow us to go directly from
> 1.4
> > to
> > > > > 1.6.
> > > > > > >
> > > > > > > In terms of a general policy, I suggest we make this sort of
> > > decision
> > > > > on
> > > > > > a
> > > > > > > case by case basis. My unreasonably self-centered intuition
> > > suggests
> > > > > that
> > > > > > > there may be some folks that want to go from 1.4 to 1.6 now due
> > to
> > > a
> > > > > > > relatively short 1.5 cycle. The need to jump multiple versions
> > like
> > > > > might
> > > > > > > not exist in the future.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> > busbey@cloudera.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > In an effort to get more users off of our now unsupported 1.4
> > > > > release,
> > > > > > > > should we support upgrading directly to 1.6 without going
> > > through a
> > > > > 1.5
> > > > > > > > upgrade?
> > > > > > > >
> > > > > > > > More directly for those on user@: would you be more likely
> to
> > > > > upgrade
> > > > > > > off
> > > > > > > > of 1.4 if you could do so directly to 1.6?
> > > > > > > >
> > > > > > > > We have this working locally at Cloudera as a part of our CDH
> > > > > > integration
> > > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > > > > > >
> > > > > > > > We can get into implementation details on a jira if there's
> > > > positive
> > > > > > > > consensus, but the changes weren't very complicated. They're
> > > mostly
> > > > > > > >
> > > > > > > > * forward porting and consolidating some upgrade code
> > > > > > > > * additions to the README for instructions
> > > > > > > >
> > > > > > > > Personally, I can see the both sides of the argument. On the
> > plus
> > > > > side,
> > > > > > > > anything to get more users off of 1.4 is a good thing. On the
> > > > > negative
> > > > > > > > side, it means we have the 1.4 related upgrade code sitting
> in
> > a
> > > > > > > supported
> > > > > > > > code branch longer.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > --
> > > > > > > > Sean
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sean
> >
>



-- 
Sean

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Christopher <ct...@apache.org>.
Cool. Thanks, Sean!


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Wed, Jun 25, 2014 at 3:25 PM, Sean Busbey <bu...@cloudera.com> wrote:

> I'll be creating a ticket and posting a patch this week.
>
>
> On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org> wrote:
>
> > So, just to revisit this conversation, it seems like there is interest in
> > supporting this. Is there already a ticket for it and/or somebody
> > interested in doing the necessary work for 1.6.1?
> >
> >
> > --
> > Christopher L Tubbs II
> > http://gravatar.com/ctubbsii
> >
> >
> > On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com> wrote:
> >
> > > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > > (ACCUMULO-2770), start 1.6
> > >
> > > Mike
> > >
> > >
> > > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <dr...@gmail.com>
> > > wrote:
> > >
> > > > Mike,
> > > >
> > > > So works just like upgrading from 1.5?
> > > >
> > > > (After 1.4 shutdown, install 1.6 and restart?)
> > > >
> > > > That sounds entirely reasonable.
> > > >
> > > > Drew
> > > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com> wrote:
> > > >
> > > > > We initially tried to set it up as a stand-alone utility but
> > eventually
> > > > > gave up. In order to properly do the upgrade, you concurrently need
> > to
> > > > run
> > > > > whatever upgrade code concurrently with a tablet server hosting
> > > !METADATA
> > > > > and a tablet server that can replay WALs. We ended up duplicating a
> > lot
> > > > of
> > > > > logic already present in master before scrapping that plan. An
> > > > alternative
> > > > > would have been to try to build on MAC, but that was also
> non-trivial
> > > to
> > > > > deploy, so we spliced the code into the existing upgrade path. How
> do
> > > you
> > > > > feel about that, Drew?
> > > > >
> > > > >
> > > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <
> drew.farris@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I'm +1 for a utility that would allow us to go directly from 1.4
> to
> > > > 1.6.
> > > > > >
> > > > > > In terms of a general policy, I suggest we make this sort of
> > decision
> > > > on
> > > > > a
> > > > > > case by case basis. My unreasonably self-centered intuition
> > suggests
> > > > that
> > > > > > there may be some folks that want to go from 1.4 to 1.6 now due
> to
> > a
> > > > > > relatively short 1.5 cycle. The need to jump multiple versions
> like
> > > > might
> > > > > > not exist in the future.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <
> busbey@cloudera.com>
> > > > > wrote:
> > > > > >
> > > > > > > In an effort to get more users off of our now unsupported 1.4
> > > > release,
> > > > > > > should we support upgrading directly to 1.6 without going
> > through a
> > > > 1.5
> > > > > > > upgrade?
> > > > > > >
> > > > > > > More directly for those on user@: would you be more likely to
> > > > upgrade
> > > > > > off
> > > > > > > of 1.4 if you could do so directly to 1.6?
> > > > > > >
> > > > > > > We have this working locally at Cloudera as a part of our CDH
> > > > > integration
> > > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > > > > >
> > > > > > > We can get into implementation details on a jira if there's
> > > positive
> > > > > > > consensus, but the changes weren't very complicated. They're
> > mostly
> > > > > > >
> > > > > > > * forward porting and consolidating some upgrade code
> > > > > > > * additions to the README for instructions
> > > > > > >
> > > > > > > Personally, I can see the both sides of the argument. On the
> plus
> > > > side,
> > > > > > > anything to get more users off of 1.4 is a good thing. On the
> > > > negative
> > > > > > > side, it means we have the 1.4 related upgrade code sitting in
> a
> > > > > > supported
> > > > > > > code branch longer.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > --
> > > > > > > Sean
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Sean
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Sean Busbey <bu...@cloudera.com>.
I'll be creating a ticket and posting a patch this week.


On Wed, Jun 25, 2014 at 2:21 PM, Christopher <ct...@apache.org> wrote:

> So, just to revisit this conversation, it seems like there is interest in
> supporting this. Is there already a ticket for it and/or somebody
> interested in doing the necessary work for 1.6.1?
>
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com> wrote:
>
> > In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> > (ACCUMULO-2770), start 1.6
> >
> > Mike
> >
> >
> > On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <dr...@gmail.com>
> > wrote:
> >
> > > Mike,
> > >
> > > So works just like upgrading from 1.5?
> > >
> > > (After 1.4 shutdown, install 1.6 and restart?)
> > >
> > > That sounds entirely reasonable.
> > >
> > > Drew
> > > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com> wrote:
> > >
> > > > We initially tried to set it up as a stand-alone utility but
> eventually
> > > > gave up. In order to properly do the upgrade, you concurrently need
> to
> > > run
> > > > whatever upgrade code concurrently with a tablet server hosting
> > !METADATA
> > > > and a tablet server that can replay WALs. We ended up duplicating a
> lot
> > > of
> > > > logic already present in master before scrapping that plan. An
> > > alternative
> > > > would have been to try to build on MAC, but that was also non-trivial
> > to
> > > > deploy, so we spliced the code into the existing upgrade path. How do
> > you
> > > > feel about that, Drew?
> > > >
> > > >
> > > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <dr...@gmail.com>
> > > > wrote:
> > > >
> > > > > I'm +1 for a utility that would allow us to go directly from 1.4 to
> > > 1.6.
> > > > >
> > > > > In terms of a general policy, I suggest we make this sort of
> decision
> > > on
> > > > a
> > > > > case by case basis. My unreasonably self-centered intuition
> suggests
> > > that
> > > > > there may be some folks that want to go from 1.4 to 1.6 now due to
> a
> > > > > relatively short 1.5 cycle. The need to jump multiple versions like
> > > might
> > > > > not exist in the future.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com>
> > > > wrote:
> > > > >
> > > > > > In an effort to get more users off of our now unsupported 1.4
> > > release,
> > > > > > should we support upgrading directly to 1.6 without going
> through a
> > > 1.5
> > > > > > upgrade?
> > > > > >
> > > > > > More directly for those on user@: would you be more likely to
> > > upgrade
> > > > > off
> > > > > > of 1.4 if you could do so directly to 1.6?
> > > > > >
> > > > > > We have this working locally at Cloudera as a part of our CDH
> > > > integration
> > > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > > > >
> > > > > > We can get into implementation details on a jira if there's
> > positive
> > > > > > consensus, but the changes weren't very complicated. They're
> mostly
> > > > > >
> > > > > > * forward porting and consolidating some upgrade code
> > > > > > * additions to the README for instructions
> > > > > >
> > > > > > Personally, I can see the both sides of the argument. On the plus
> > > side,
> > > > > > anything to get more users off of 1.4 is a good thing. On the
> > > negative
> > > > > > side, it means we have the 1.4 related upgrade code sitting in a
> > > > > supported
> > > > > > code branch longer.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > --
> > > > > > Sean
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Sean

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Christopher <ct...@apache.org>.
So, just to revisit this conversation, it seems like there is interest in
supporting this. Is there already a ticket for it and/or somebody
interested in doing the necessary work for 1.6.1?


--
Christopher L Tubbs II
http://gravatar.com/ctubbsii


On Thu, Jun 19, 2014 at 11:07 AM, Mike Drob <ma...@cloudera.com> wrote:

> In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
> (ACCUMULO-2770), start 1.6
>
> Mike
>
>
> On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <dr...@gmail.com>
> wrote:
>
> > Mike,
> >
> > So works just like upgrading from 1.5?
> >
> > (After 1.4 shutdown, install 1.6 and restart?)
> >
> > That sounds entirely reasonable.
> >
> > Drew
> > On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com> wrote:
> >
> > > We initially tried to set it up as a stand-alone utility but eventually
> > > gave up. In order to properly do the upgrade, you concurrently need to
> > run
> > > whatever upgrade code concurrently with a tablet server hosting
> !METADATA
> > > and a tablet server that can replay WALs. We ended up duplicating a lot
> > of
> > > logic already present in master before scrapping that plan. An
> > alternative
> > > would have been to try to build on MAC, but that was also non-trivial
> to
> > > deploy, so we spliced the code into the existing upgrade path. How do
> you
> > > feel about that, Drew?
> > >
> > >
> > > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <dr...@gmail.com>
> > > wrote:
> > >
> > > > I'm +1 for a utility that would allow us to go directly from 1.4 to
> > 1.6.
> > > >
> > > > In terms of a general policy, I suggest we make this sort of decision
> > on
> > > a
> > > > case by case basis. My unreasonably self-centered intuition suggests
> > that
> > > > there may be some folks that want to go from 1.4 to 1.6 now due to a
> > > > relatively short 1.5 cycle. The need to jump multiple versions like
> > might
> > > > not exist in the future.
> > > >
> > > >
> > > >
> > > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com>
> > > wrote:
> > > >
> > > > > In an effort to get more users off of our now unsupported 1.4
> > release,
> > > > > should we support upgrading directly to 1.6 without going through a
> > 1.5
> > > > > upgrade?
> > > > >
> > > > > More directly for those on user@: would you be more likely to
> > upgrade
> > > > off
> > > > > of 1.4 if you could do so directly to 1.6?
> > > > >
> > > > > We have this working locally at Cloudera as a part of our CDH
> > > integration
> > > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > > >
> > > > > We can get into implementation details on a jira if there's
> positive
> > > > > consensus, but the changes weren't very complicated. They're mostly
> > > > >
> > > > > * forward porting and consolidating some upgrade code
> > > > > * additions to the README for instructions
> > > > >
> > > > > Personally, I can see the both sides of the argument. On the plus
> > side,
> > > > > anything to get more users off of 1.4 is a good thing. On the
> > negative
> > > > > side, it means we have the 1.4 related upgrade code sitting in a
> > > > supported
> > > > > code branch longer.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > --
> > > > > Sean
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Mike Drob <ma...@cloudera.com>.
In a nutshell: stop 1.4, install 1.6, copy the WALs to HDFS
(ACCUMULO-2770), start 1.6

Mike


On Wed, Jun 18, 2014 at 5:54 PM, Drew Farris <dr...@gmail.com> wrote:

> Mike,
>
> So works just like upgrading from 1.5?
>
> (After 1.4 shutdown, install 1.6 and restart?)
>
> That sounds entirely reasonable.
>
> Drew
> On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com> wrote:
>
> > We initially tried to set it up as a stand-alone utility but eventually
> > gave up. In order to properly do the upgrade, you concurrently need to
> run
> > whatever upgrade code concurrently with a tablet server hosting !METADATA
> > and a tablet server that can replay WALs. We ended up duplicating a lot
> of
> > logic already present in master before scrapping that plan. An
> alternative
> > would have been to try to build on MAC, but that was also non-trivial to
> > deploy, so we spliced the code into the existing upgrade path. How do you
> > feel about that, Drew?
> >
> >
> > On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <dr...@gmail.com>
> > wrote:
> >
> > > I'm +1 for a utility that would allow us to go directly from 1.4 to
> 1.6.
> > >
> > > In terms of a general policy, I suggest we make this sort of decision
> on
> > a
> > > case by case basis. My unreasonably self-centered intuition suggests
> that
> > > there may be some folks that want to go from 1.4 to 1.6 now due to a
> > > relatively short 1.5 cycle. The need to jump multiple versions like
> might
> > > not exist in the future.
> > >
> > >
> > >
> > > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > > > In an effort to get more users off of our now unsupported 1.4
> release,
> > > > should we support upgrading directly to 1.6 without going through a
> 1.5
> > > > upgrade?
> > > >
> > > > More directly for those on user@: would you be more likely to
> upgrade
> > > off
> > > > of 1.4 if you could do so directly to 1.6?
> > > >
> > > > We have this working locally at Cloudera as a part of our CDH
> > integration
> > > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > > >
> > > > We can get into implementation details on a jira if there's positive
> > > > consensus, but the changes weren't very complicated. They're mostly
> > > >
> > > > * forward porting and consolidating some upgrade code
> > > > * additions to the README for instructions
> > > >
> > > > Personally, I can see the both sides of the argument. On the plus
> side,
> > > > anything to get more users off of 1.4 is a good thing. On the
> negative
> > > > side, it means we have the 1.4 related upgrade code sitting in a
> > > supported
> > > > code branch longer.
> > > >
> > > > Thoughts?
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> >
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Drew Farris <dr...@gmail.com>.
Mike,

So works just like upgrading from 1.5?

(After 1.4 shutdown, install 1.6 and restart?)

That sounds entirely reasonable.

Drew
On Jun 17, 2014 10:52 PM, "Mike Drob" <ma...@cloudera.com> wrote:

> We initially tried to set it up as a stand-alone utility but eventually
> gave up. In order to properly do the upgrade, you concurrently need to run
> whatever upgrade code concurrently with a tablet server hosting !METADATA
> and a tablet server that can replay WALs. We ended up duplicating a lot of
> logic already present in master before scrapping that plan. An alternative
> would have been to try to build on MAC, but that was also non-trivial to
> deploy, so we spliced the code into the existing upgrade path. How do you
> feel about that, Drew?
>
>
> On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <dr...@gmail.com>
> wrote:
>
> > I'm +1 for a utility that would allow us to go directly from 1.4 to 1.6.
> >
> > In terms of a general policy, I suggest we make this sort of decision on
> a
> > case by case basis. My unreasonably self-centered intuition suggests that
> > there may be some folks that want to go from 1.4 to 1.6 now due to a
> > relatively short 1.5 cycle. The need to jump multiple versions like might
> > not exist in the future.
> >
> >
> >
> > On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > In an effort to get more users off of our now unsupported 1.4 release,
> > > should we support upgrading directly to 1.6 without going through a 1.5
> > > upgrade?
> > >
> > > More directly for those on user@: would you be more likely to upgrade
> > off
> > > of 1.4 if you could do so directly to 1.6?
> > >
> > > We have this working locally at Cloudera as a part of our CDH
> integration
> > > (we shipped 1.4 and we're planning to ship 1.6 next).
> > >
> > > We can get into implementation details on a jira if there's positive
> > > consensus, but the changes weren't very complicated. They're mostly
> > >
> > > * forward porting and consolidating some upgrade code
> > > * additions to the README for instructions
> > >
> > > Personally, I can see the both sides of the argument. On the plus side,
> > > anything to get more users off of 1.4 is a good thing. On the negative
> > > side, it means we have the 1.4 related upgrade code sitting in a
> > supported
> > > code branch longer.
> > >
> > > Thoughts?
> > >
> > > --
> > > Sean
> > >
> >
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Mike Drob <ma...@cloudera.com>.
We initially tried to set it up as a stand-alone utility but eventually
gave up. In order to properly do the upgrade, you concurrently need to run
whatever upgrade code concurrently with a tablet server hosting !METADATA
and a tablet server that can replay WALs. We ended up duplicating a lot of
logic already present in master before scrapping that plan. An alternative
would have been to try to build on MAC, but that was also non-trivial to
deploy, so we spliced the code into the existing upgrade path. How do you
feel about that, Drew?


On Tue, Jun 17, 2014 at 8:57 PM, Drew Farris <dr...@gmail.com> wrote:

> I'm +1 for a utility that would allow us to go directly from 1.4 to 1.6.
>
> In terms of a general policy, I suggest we make this sort of decision on a
> case by case basis. My unreasonably self-centered intuition suggests that
> there may be some folks that want to go from 1.4 to 1.6 now due to a
> relatively short 1.5 cycle. The need to jump multiple versions like might
> not exist in the future.
>
>
>
> On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > In an effort to get more users off of our now unsupported 1.4 release,
> > should we support upgrading directly to 1.6 without going through a 1.5
> > upgrade?
> >
> > More directly for those on user@: would you be more likely to upgrade
> off
> > of 1.4 if you could do so directly to 1.6?
> >
> > We have this working locally at Cloudera as a part of our CDH integration
> > (we shipped 1.4 and we're planning to ship 1.6 next).
> >
> > We can get into implementation details on a jira if there's positive
> > consensus, but the changes weren't very complicated. They're mostly
> >
> > * forward porting and consolidating some upgrade code
> > * additions to the README for instructions
> >
> > Personally, I can see the both sides of the argument. On the plus side,
> > anything to get more users off of 1.4 is a good thing. On the negative
> > side, it means we have the 1.4 related upgrade code sitting in a
> supported
> > code branch longer.
> >
> > Thoughts?
> >
> > --
> > Sean
> >
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Drew Farris <dr...@gmail.com>.
I'm +1 for a utility that would allow us to go directly from 1.4 to 1.6.

In terms of a general policy, I suggest we make this sort of decision on a
case by case basis. My unreasonably self-centered intuition suggests that
there may be some folks that want to go from 1.4 to 1.6 now due to a
relatively short 1.5 cycle. The need to jump multiple versions like might
not exist in the future.



On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:

> In an effort to get more users off of our now unsupported 1.4 release,
> should we support upgrading directly to 1.6 without going through a 1.5
> upgrade?
>
> More directly for those on user@: would you be more likely to upgrade off
> of 1.4 if you could do so directly to 1.6?
>
> We have this working locally at Cloudera as a part of our CDH integration
> (we shipped 1.4 and we're planning to ship 1.6 next).
>
> We can get into implementation details on a jira if there's positive
> consensus, but the changes weren't very complicated. They're mostly
>
> * forward porting and consolidating some upgrade code
> * additions to the README for instructions
>
> Personally, I can see the both sides of the argument. On the plus side,
> anything to get more users off of 1.4 is a good thing. On the negative
> side, it means we have the 1.4 related upgrade code sitting in a supported
> code branch longer.
>
> Thoughts?
>
> --
> Sean
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by William Slacum <wi...@accumulo.net>.
How much of this is a standalone utility? I think a magic button approach
would be good for this case.


On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:

> In an effort to get more users off of our now unsupported 1.4 release,
> should we support upgrading directly to 1.6 without going through a 1.5
> upgrade?
>
> More directly for those on user@: would you be more likely to upgrade off
> of 1.4 if you could do so directly to 1.6?
>
> We have this working locally at Cloudera as a part of our CDH integration
> (we shipped 1.4 and we're planning to ship 1.6 next).
>
> We can get into implementation details on a jira if there's positive
> consensus, but the changes weren't very complicated. They're mostly
>
> * forward porting and consolidating some upgrade code
> * additions to the README for instructions
>
> Personally, I can see the both sides of the argument. On the plus side,
> anything to get more users off of 1.4 is a good thing. On the negative
> side, it means we have the 1.4 related upgrade code sitting in a supported
> code branch longer.
>
> Thoughts?
>
> --
> Sean
>

Re: [DISCUSS] Should we support upgrading 1.4 -> 1.6 w/o going through 1.5?

Posted by Drew Farris <dr...@gmail.com>.
I'm +1 for a utility that would allow us to go directly from 1.4 to 1.6.

In terms of a general policy, I suggest we make this sort of decision on a
case by case basis. My unreasonably self-centered intuition suggests that
there may be some folks that want to go from 1.4 to 1.6 now due to a
relatively short 1.5 cycle. The need to jump multiple versions like might
not exist in the future.



On Mon, Jun 16, 2014 at 5:24 PM, Sean Busbey <bu...@cloudera.com> wrote:

> In an effort to get more users off of our now unsupported 1.4 release,
> should we support upgrading directly to 1.6 without going through a 1.5
> upgrade?
>
> More directly for those on user@: would you be more likely to upgrade off
> of 1.4 if you could do so directly to 1.6?
>
> We have this working locally at Cloudera as a part of our CDH integration
> (we shipped 1.4 and we're planning to ship 1.6 next).
>
> We can get into implementation details on a jira if there's positive
> consensus, but the changes weren't very complicated. They're mostly
>
> * forward porting and consolidating some upgrade code
> * additions to the README for instructions
>
> Personally, I can see the both sides of the argument. On the plus side,
> anything to get more users off of 1.4 is a good thing. On the negative
> side, it means we have the 1.4 related upgrade code sitting in a supported
> code branch longer.
>
> Thoughts?
>
> --
> Sean
>