deb versioned database dirs

[Randall Leeds <ra...@gmail.com>]

The deb packaging I pulled from downstream has the following:

"
Apache CouchDB is alpha software and still under heavy development. Please be
aware that important areas such as the public API or internal database format
may see backwards incompatible changes between versions.

...

The partitioned database directories are named like this:

  /var/lib/couchdb/VERSION

...

 -- Sam Bisbee <sb...@computervip.com>  Wed, 11 Nov 2009 23:22:21 -0500
"

I'm considering removing this since it seems outdated and reverting to
the normal directories. What do you all think?

-Randall

Re: deb versioned database dirs

[Robert Newson <rn...@apache.org>]

+1.

On 6 April 2012 22:52, Randall Leeds <ra...@gmail.com> wrote:
> The deb packaging I pulled from downstream has the following:
>
> "
> Apache CouchDB is alpha software and still under heavy development. Please be
> aware that important areas such as the public API or internal database format
> may see backwards incompatible changes between versions.
>
> ...
>
> The partitioned database directories are named like this:
>
>  /var/lib/couchdb/VERSION
>
> ...
>
>  -- Sam Bisbee <sb...@computervip.com>  Wed, 11 Nov 2009 23:22:21 -0500
> "
>
> I'm considering removing this since it seems outdated and reverting to
> the normal directories. What do you all think?
>
> -Randall

Re: deb versioned database dirs

[Noah Slater <ns...@tumbolia.org>]

Okay, cool. :)

On Sat, Sep 15, 2012 at 12:06 AM, Randall Leeds <ra...@gmail.com>wrote:

> On Wed, Sep 12, 2012 at 3:48 AM, Robert Newson <rn...@apache.org> wrote:
> >
> > It's currently the case, yes. We can read 0.10.0 onwards, I think.
> >
> > What I think we *can* promise is that we'll read 1.0.0 onwards for the
> forseeable future. If that changes, then that upgrade is the one that will
> need extra care and attention.
>
> I tend to agree. The common case should be compatibility. We can alert
> and require intervention for some future version if and when that
> becomes necessary.
>
> If we're okay with this approach, I'll update my packaging and publish
> the work I've done for the rest who are interested in debian
> packaging. Then I will update my PPA.
>
> -Randall
>



-- 
NS

Re: deb versioned database dirs

[Randall Leeds <ra...@gmail.com>]

On Wed, Sep 12, 2012 at 3:48 AM, Robert Newson <rn...@apache.org> wrote:
>
> It's currently the case, yes. We can read 0.10.0 onwards, I think.
>
> What I think we *can* promise is that we'll read 1.0.0 onwards for the forseeable future. If that changes, then that upgrade is the one that will need extra care and attention.

I tend to agree. The common case should be compatibility. We can alert
and require intervention for some future version if and when that
becomes necessary.

If we're okay with this approach, I'll update my packaging and publish
the work I've done for the rest who are interested in debian
packaging. Then I will update my PPA.

-Randall

Re: deb versioned database dirs

[Robert Newson <rn...@apache.org>]

It's currently the case, yes. We can read 0.10.0 onwards, I think.

What I think we *can* promise is that we'll read 1.0.0 onwards for the forseeable future. If that changes, then that upgrade is the one that will need extra care and attention.

B.

On 12 Sep 2012, at 11:36, Noah Slater wrote:

> In my experience of Debian, when you upgrade Postgres or MySQL from one
> version to another, you actually have different packages. So postgres2.3,
> postgres2.4, etc. Which means, because of how Debian works, you're getting
> different directories right out of the box. (Though, they may share the
> same database files, anyway. Not sure. Would have to check.)
> 
> I can't actually remember whether it is done on major revisions, or minor
> revisions. It probably differs by package. I know Emacs, for example, uses
> minor revisions in the package name.
> 
> I that system, an upgrade between point releases is done with the same
> package, and is done transparently and automatically. To upgrade a major
> (or minor, depending on the system) version, you have to install a new
> package.
> 
> Would that fit our model?
> 
> Bob, are you saying that we will never fail to read old formats post-1.0?
> That seems like quote a boast, and was something I was unaware of until
> now. I guess that changes our approach considerably.
> 
> On Wed, Sep 12, 2012 at 10:38 AM, Robert Newson <rn...@apache.org> wrote:
> 
>> 
>> From 1.0.0 onwards we promise to support forward migration of database and
>> view files, we shouldn't be introducing any breaking changes that prevent
>> any future couchdb release from reading .couch files from 1.0.0 or any
>> release hence. The versioned database dir pattern must have been useful in
>> the past, but it's obsolete now. We could look to see what other db's do
>> when an administrator upgrades to a new version that also changes the
>> on-disk format, especially since we don't provide a downgrade path.
>> 
>> One thought is that we disable or remove the automatic upgrading to the
>> new format and allow administrators to do it electively. It's a fair amount
>> of work, though. I would rather show a warning for those upgrades that
>> change disk_version that informs the administrator and recommends taking a
>> backup before proceeding.
>> 
>> B.
>> 
>> On 12 Sep 2012, at 05:25, Randall Leeds wrote:
>> 
>>> On Sun, Sep 9, 2012 at 6:40 AM, Noah Slater <ns...@tumbolia.org>
>> wrote:
>>>> Randall,
>>>> 
>>>> Sorry for the very late reply (doing some long overdue email sprints).
>>> 
>>> Absolutely no worries. I've been doing a bit of the same myself. And
>>> if you hadn't noticed, I haven't been hacking on couch much in the
>>> last several months, sadly.
>>> 
>>>> 
>>>> What did you do about this in the end?
>>> 
>>> So far, my hacking has resulted in a package that alerts the user
>>> (with a menu in the standard debian way) that they will need to
>>> migrate the database files by hand after the upgrade. It then installs
>>> CouchDB with /var/lib/couchdb as the database directory (rather than
>>> /var/lib/couchdb/X.Y.Z). This package is in my PPA[0].
>>> 
>>>> 
>>>> I was the one who designed that scheme for Debian, so I can, at least,
>> let
>>>> you know the thinking behind it.
>>>> 
>>>> I was concerned about an update to CouchDB breaking, or corrupting, old
>>>> data. So, new package versions, obviously, install a fresh CouchDB, and
>> it
>>>> is up to you to port the data across.
>>> 
>>> That's totally clear. It's what I had guessed and I think it was the
>>> right decision, for sure, at that time.
>>> 
>>>> 
>>>> Now, CouchDB is a lot more stable than it used to be, but we do, still,
>>>> have breaking changes from time to time.
>>>> 
>>>> How would those be handled by your package?
>>>> 
>>>> (Or were you just installing the package, and not hacking on it, in
>> which
>>>> case, should we file a bug downstream?)
>>> 
>>> My thinking was that the common case these days in not breaking
>>> changes, or at least breaking changes for which the software manages
>>> upgrades on compaction and reads the old format.
>>> 
>>> With your feedback, and other devs, I'd like to know what our approach
>>> should be. Should we continue to version the database directory at
>>> all? Should we use major versions only (/var/lib/couchdb/{1,2}.x)?
>>> Should we keep the old scheme?
>>> 
>>> The problem with always requiring a manual migration is that CouchDB
>>> may be updated and then restarted without human intervention.
>>> Suddenly, then, it appears to have no data! *If* we make a commitment
>>> to support and upgrade database files in place, at least through a
>>> single major version, then this problem can be avoided for many
>>> upgrades. For example, Ubuntu could include our package verbatim in
>>> the repos and know that they can ship security updates that only
>>> increment the revision and users can install CouchDB on Ubuntu server
>>> and trust the system to perform security updates without their
>>> database disappearing suddenly.
>>> 
>>> I/we can do this any way we please, we just need to decide and justify
>>> our decision.
>>> 
>>> Right now, I think I'd lean towards major version directories, or
>>> perhaps instead disabling the init service via a flag in
>>> /etc/default/couchdb on major version updates to force user
>>> intervention only when necessary.
>>> 
>>> I'd love your thoughts.
>>> 
>>> [0] https://launchpad.net/~randall-leeds/+archive/couchdb
>> 
>> 
> 
> 
> -- 
> NS

Re: deb versioned database dirs

[Noah Slater <ns...@tumbolia.org>]

In my experience of Debian, when you upgrade Postgres or MySQL from one
version to another, you actually have different packages. So postgres2.3,
postgres2.4, etc. Which means, because of how Debian works, you're getting
different directories right out of the box. (Though, they may share the
same database files, anyway. Not sure. Would have to check.)

I can't actually remember whether it is done on major revisions, or minor
revisions. It probably differs by package. I know Emacs, for example, uses
minor revisions in the package name.

I that system, an upgrade between point releases is done with the same
package, and is done transparently and automatically. To upgrade a major
(or minor, depending on the system) version, you have to install a new
package.

Would that fit our model?

Bob, are you saying that we will never fail to read old formats post-1.0?
That seems like quote a boast, and was something I was unaware of until
now. I guess that changes our approach considerably.

On Wed, Sep 12, 2012 at 10:38 AM, Robert Newson <rn...@apache.org> wrote:

>
> From 1.0.0 onwards we promise to support forward migration of database and
> view files, we shouldn't be introducing any breaking changes that prevent
> any future couchdb release from reading .couch files from 1.0.0 or any
> release hence. The versioned database dir pattern must have been useful in
> the past, but it's obsolete now. We could look to see what other db's do
> when an administrator upgrades to a new version that also changes the
> on-disk format, especially since we don't provide a downgrade path.
>
> One thought is that we disable or remove the automatic upgrading to the
> new format and allow administrators to do it electively. It's a fair amount
> of work, though. I would rather show a warning for those upgrades that
> change disk_version that informs the administrator and recommends taking a
> backup before proceeding.
>
> B.
>
> On 12 Sep 2012, at 05:25, Randall Leeds wrote:
>
> > On Sun, Sep 9, 2012 at 6:40 AM, Noah Slater <ns...@tumbolia.org>
> wrote:
> >> Randall,
> >>
> >> Sorry for the very late reply (doing some long overdue email sprints).
> >
> > Absolutely no worries. I've been doing a bit of the same myself. And
> > if you hadn't noticed, I haven't been hacking on couch much in the
> > last several months, sadly.
> >
> >>
> >> What did you do about this in the end?
> >
> > So far, my hacking has resulted in a package that alerts the user
> > (with a menu in the standard debian way) that they will need to
> > migrate the database files by hand after the upgrade. It then installs
> > CouchDB with /var/lib/couchdb as the database directory (rather than
> > /var/lib/couchdb/X.Y.Z). This package is in my PPA[0].
> >
> >>
> >> I was the one who designed that scheme for Debian, so I can, at least,
> let
> >> you know the thinking behind it.
> >>
> >> I was concerned about an update to CouchDB breaking, or corrupting, old
> >> data. So, new package versions, obviously, install a fresh CouchDB, and
> it
> >> is up to you to port the data across.
> >
> > That's totally clear. It's what I had guessed and I think it was the
> > right decision, for sure, at that time.
> >
> >>
> >> Now, CouchDB is a lot more stable than it used to be, but we do, still,
> >> have breaking changes from time to time.
> >>
> >> How would those be handled by your package?
> >>
> >> (Or were you just installing the package, and not hacking on it, in
> which
> >> case, should we file a bug downstream?)
> >
> > My thinking was that the common case these days in not breaking
> > changes, or at least breaking changes for which the software manages
> > upgrades on compaction and reads the old format.
> >
> > With your feedback, and other devs, I'd like to know what our approach
> > should be. Should we continue to version the database directory at
> > all? Should we use major versions only (/var/lib/couchdb/{1,2}.x)?
> > Should we keep the old scheme?
> >
> > The problem with always requiring a manual migration is that CouchDB
> > may be updated and then restarted without human intervention.
> > Suddenly, then, it appears to have no data! *If* we make a commitment
> > to support and upgrade database files in place, at least through a
> > single major version, then this problem can be avoided for many
> > upgrades. For example, Ubuntu could include our package verbatim in
> > the repos and know that they can ship security updates that only
> > increment the revision and users can install CouchDB on Ubuntu server
> > and trust the system to perform security updates without their
> > database disappearing suddenly.
> >
> > I/we can do this any way we please, we just need to decide and justify
> > our decision.
> >
> > Right now, I think I'd lean towards major version directories, or
> > perhaps instead disabling the init service via a flag in
> > /etc/default/couchdb on major version updates to force user
> > intervention only when necessary.
> >
> > I'd love your thoughts.
> >
> > [0] https://launchpad.net/~randall-leeds/+archive/couchdb
>
>


-- 
NS

Re: deb versioned database dirs

[Robert Newson <rn...@apache.org>]

From 1.0.0 onwards we promise to support forward migration of database and view files, we shouldn't be introducing any breaking changes that prevent any future couchdb release from reading .couch files from 1.0.0 or any release hence. The versioned database dir pattern must have been useful in the past, but it's obsolete now. We could look to see what other db's do when an administrator upgrades to a new version that also changes the on-disk format, especially since we don't provide a downgrade path.

One thought is that we disable or remove the automatic upgrading to the new format and allow administrators to do it electively. It's a fair amount of work, though. I would rather show a warning for those upgrades that change disk_version that informs the administrator and recommends taking a backup before proceeding.

B.

On 12 Sep 2012, at 05:25, Randall Leeds wrote:

> On Sun, Sep 9, 2012 at 6:40 AM, Noah Slater <ns...@tumbolia.org> wrote:
>> Randall,
>> 
>> Sorry for the very late reply (doing some long overdue email sprints).
> 
> Absolutely no worries. I've been doing a bit of the same myself. And
> if you hadn't noticed, I haven't been hacking on couch much in the
> last several months, sadly.
> 
>> 
>> What did you do about this in the end?
> 
> So far, my hacking has resulted in a package that alerts the user
> (with a menu in the standard debian way) that they will need to
> migrate the database files by hand after the upgrade. It then installs
> CouchDB with /var/lib/couchdb as the database directory (rather than
> /var/lib/couchdb/X.Y.Z). This package is in my PPA[0].
> 
>> 
>> I was the one who designed that scheme for Debian, so I can, at least, let
>> you know the thinking behind it.
>> 
>> I was concerned about an update to CouchDB breaking, or corrupting, old
>> data. So, new package versions, obviously, install a fresh CouchDB, and it
>> is up to you to port the data across.
> 
> That's totally clear. It's what I had guessed and I think it was the
> right decision, for sure, at that time.
> 
>> 
>> Now, CouchDB is a lot more stable than it used to be, but we do, still,
>> have breaking changes from time to time.
>> 
>> How would those be handled by your package?
>> 
>> (Or were you just installing the package, and not hacking on it, in which
>> case, should we file a bug downstream?)
> 
> My thinking was that the common case these days in not breaking
> changes, or at least breaking changes for which the software manages
> upgrades on compaction and reads the old format.
> 
> With your feedback, and other devs, I'd like to know what our approach
> should be. Should we continue to version the database directory at
> all? Should we use major versions only (/var/lib/couchdb/{1,2}.x)?
> Should we keep the old scheme?
> 
> The problem with always requiring a manual migration is that CouchDB
> may be updated and then restarted without human intervention.
> Suddenly, then, it appears to have no data! *If* we make a commitment
> to support and upgrade database files in place, at least through a
> single major version, then this problem can be avoided for many
> upgrades. For example, Ubuntu could include our package verbatim in
> the repos and know that they can ship security updates that only
> increment the revision and users can install CouchDB on Ubuntu server
> and trust the system to perform security updates without their
> database disappearing suddenly.
> 
> I/we can do this any way we please, we just need to decide and justify
> our decision.
> 
> Right now, I think I'd lean towards major version directories, or
> perhaps instead disabling the init service via a flag in
> /etc/default/couchdb on major version updates to force user
> intervention only when necessary.
> 
> I'd love your thoughts.
> 
> [0] https://launchpad.net/~randall-leeds/+archive/couchdb

Re: deb versioned database dirs

[Randall Leeds <ra...@gmail.com>]

On Sun, Sep 9, 2012 at 6:40 AM, Noah Slater <ns...@tumbolia.org> wrote:
> Randall,
>
> Sorry for the very late reply (doing some long overdue email sprints).

Absolutely no worries. I've been doing a bit of the same myself. And
if you hadn't noticed, I haven't been hacking on couch much in the
last several months, sadly.

>
> What did you do about this in the end?

So far, my hacking has resulted in a package that alerts the user
(with a menu in the standard debian way) that they will need to
migrate the database files by hand after the upgrade. It then installs
CouchDB with /var/lib/couchdb as the database directory (rather than
/var/lib/couchdb/X.Y.Z). This package is in my PPA[0].

>
> I was the one who designed that scheme for Debian, so I can, at least, let
> you know the thinking behind it.
>
> I was concerned about an update to CouchDB breaking, or corrupting, old
> data. So, new package versions, obviously, install a fresh CouchDB, and it
> is up to you to port the data across.

That's totally clear. It's what I had guessed and I think it was the
right decision, for sure, at that time.

>
> Now, CouchDB is a lot more stable than it used to be, but we do, still,
> have breaking changes from time to time.
>
> How would those be handled by your package?
>
> (Or were you just installing the package, and not hacking on it, in which
> case, should we file a bug downstream?)

My thinking was that the common case these days in not breaking
changes, or at least breaking changes for which the software manages
upgrades on compaction and reads the old format.

With your feedback, and other devs, I'd like to know what our approach
should be. Should we continue to version the database directory at
all? Should we use major versions only (/var/lib/couchdb/{1,2}.x)?
Should we keep the old scheme?

The problem with always requiring a manual migration is that CouchDB
may be updated and then restarted without human intervention.
Suddenly, then, it appears to have no data! *If* we make a commitment
to support and upgrade database files in place, at least through a
single major version, then this problem can be avoided for many
upgrades. For example, Ubuntu could include our package verbatim in
the repos and know that they can ship security updates that only
increment the revision and users can install CouchDB on Ubuntu server
and trust the system to perform security updates without their
database disappearing suddenly.

I/we can do this any way we please, we just need to decide and justify
our decision.

Right now, I think I'd lean towards major version directories, or
perhaps instead disabling the init service via a flag in
/etc/default/couchdb on major version updates to force user
intervention only when necessary.

I'd love your thoughts.

[0] https://launchpad.net/~randall-leeds/+archive/couchdb

Re: deb versioned database dirs

[Noah Slater <ns...@tumbolia.org>]

Randall,

Sorry for the very late reply (doing some long overdue email sprints).

What did you do about this in the end?

I was the one who designed that scheme for Debian, so I can, at least, let
you know the thinking behind it.

I was concerned about an update to CouchDB breaking, or corrupting, old
data. So, new package versions, obviously, install a fresh CouchDB, and it
is up to you to port the data across.

Now, CouchDB is a lot more stable than it used to be, but we do, still,
have breaking changes from time to time.

How would those be handled by your package?

(Or were you just installing the package, and not hacking on it, in which
case, should we file a bug downstream?)

On Sat, Apr 7, 2012 at 3:52 AM, Randall Leeds <ra...@gmail.com>wrote:

> The deb packaging I pulled from downstream has the following:
>
> "
> Apache CouchDB is alpha software and still under heavy development. Please
> be
> aware that important areas such as the public API or internal database
> format
> may see backwards incompatible changes between versions.
>
> ...
>
> The partitioned database directories are named like this:
>
>   /var/lib/couchdb/VERSION
>
> ...
>
>  -- Sam Bisbee <sb...@computervip.com>  Wed, 11 Nov 2009 23:22:21 -0500
> "
>
> I'm considering removing this since it seems outdated and reverting to
> the normal directories. What do you all think?
>
> -Randall
>



-- 
NS