You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Rick Hillegas <Ri...@Sun.COM> on 2006/03/20 21:56:51 UTC
Soft Upgrade and Downgrade
I'm looking for an explanation of how Soft Upgrade is supposed to work,
from the customer's perspective and from the developer's perspective. I
suppose that this also entails an explanation of the inverse operation,
Soft Downgrade. I can't find this explanation.
My naive understanding of soft upgrade is that we do not want to require
that server code and data be at the same rev level. Instead, data at a
given rev level should be readable and writable by all subsequent revs
of the server. But this a vague description. It is apparently ok for
advanced features to not work against down-rev data. I can't figure out
what limitations are considered acceptable and how those limitations
should manifest themselves to customers. Are there general rules or is
this decided on a release-by-release basis?
1) What is the customer-visible behavior of Soft Upgrade and Soft
Downgrade? The 10.1 Release Notes are silent on this topic. The 10.2
Developer's Guide lists 10.2 features which won't work on softly
upgraded data but is not explicit about what happens if you run a 10.0
server against softly upgraded data. It just says that you can run a
10.0 server against softly upgraded data.
2) Does Soft Upgrade in fact change the shape or content of any data in
the database? If so, where do we collect the rules about what changes
are allowed?
3) It appears that we expect to be able to run down-rev clients against
up-rev servers running against down-rev data. For the most advanced
server rev, the clients can range over all previous versions and so can
the data. If there are N previous versions, Soft Upgrade creates N*N
additional code paths which we have to test.
4) What is the customer problem solved by Soft Upgrade? Why can't we
require that the server and its data be at the same rev level?
Thanks,
-Rick
Re: Soft Upgrade and Downgrade
Posted by Oystein Grovlen - Sun Norway <Oy...@Sun.COM>.
Rick Hillegas wrote:
> 3) It appears that we expect to be able to run down-rev clients against
> up-rev servers running against down-rev data. For the most advanced
> server rev, the clients can range over all previous versions and so can
> the data. If there are N previous versions, Soft Upgrade creates N*N
> additional code paths which we have to test.
Is there really a need to provide soft upgrade from one release to any
newer release? Would it be possible to limit these code paths my saying
that soft upgrade is not only provided to the next major release. E.g,
in order to upgrade from 10.0 to 10.2, one would have to go through the
following steps:
1. Do a soft upgrade to 10.1
2. Verify that everything works and there is no reason to revert to 10.0
3. Do a hard upgrade to 10.1
4. Do a soft upgrade to 10.2
>
> 4) What is the customer problem solved by Soft Upgrade? Why can't we
> require that the server and its data be at the same rev level?
>
As far as I have understood, soft upgrade is used in order to be able to
downgrade back to the old version.
--
Øystein
Re: Soft Upgrade and Downgrade
Posted by Oystein Grovlen - Sun Norway <Oy...@Sun.COM>.
Mike Matrigali wrote:
> The way I look at it, derby has a very nice default. We call it soft
> upgrade, but in my mind it really is "no upgrade". We guarantee that
> if you do nothing to your existing app, then you can run it and the
> database against your current version of the derby software and all
> future versions of the derby software. I think this is a reasonable
> "automatic" default.
I do not agree that it is "no upgrade". The Derby code is upgraded, and
any Derby "customer" needs to validate this new version the same way as
they regularly when replacing software components, regardless of whether
the upgrade is hard or soft.
--
Øystein
Re: Soft Upgrade and Downgrade
Posted by Satheesh Bandaram <sa...@Sourcery.Org>.
Good points... Also, if Derby is being embedded in other applications
that provide reverting back to older versions, wouldn't automatic
upgrade of Derby prevent embedded application from reverting? I do think
soft-upgrade is a very useful feature to have, especially for embedded
databases.
Satheesh
Mike Matrigali wrote:
> The way I look at it, derby has a very nice default. We call it soft
> upgrade, but in my mind it really is "no upgrade". We guarantee that
> if you do nothing to your existing app, then you can run it and the
> database against your current version of the derby software and all
> future versions of the derby software. I think this is a reasonable
> "automatic" default.
>
> Because we provide no way to "go back" with database versions I believe
> it is reasonable to require the very slight user intervention of
> connecting and requiring the upgrade=true flag. If you want to use new
> stuff then you proactively tell us. This model is really important if
> we ever decide to do some sort of major data upgrade where the cost
> of hard upgrade may be hours depending on the size of the db and number
> of rows touched.
>
> Oystein Grovlen - Sun Norway wrote:
>
>> I agree that soft upgrade has a value, but I think you give soft
>> upgrade credit for things that are actual not provided by soft
>> upgrade. More comments below.
>>
>> Kathey Marsden wrote:
>>
>>> Rick Hillegas wrote:
>>>
>>>> 4) What is the customer problem solved by Soft Upgrade?
>>>
>>>
>>>
>>>
>>> Soft upgrade solves many problems for users , support and
>>> development. It:
>>>
>>> - Minimizes end user intervention in accordance with Derby's charter
>>> http://db.apache.org/derby/derby_charter.html.
>>> which calls for ease of use and zero administration for end users.
>>
>>
>>
>> It is not soft upgrade that minimizes end user intervention. That is
>> provided by automatic upgrade (soft or hard). In fact, soft upgrade
>> increases user intervention since you will need to explicitly state
>> that you want to do a hard upgrade. Without soft upgrade, hard
>> upgrades could have been done automatically.
>>
>>> - Encourages upgrade and reduces the need to port fixes to old
>>> releases.
>>
>>
>>
>> I agree that people will probably be less hesistant to upgrade to new
>> releases if they are able to revert to their old release. In my
>> opinion, that is the only value of soft upgrade. In itself, I would
>> not think that using the old data format should make it much less
>> risky to upgrade than using a new data format. And if I had to
>> downgrade due to problems with the new release, how do I know that
>> this new faulty release has not corrupted my database?
>>
>>> - Provides applications, embedding or including Derby a scalable
>>> deployment option for distributing new versions of Derby.
>>
>>
>>
>> How is soft upgrade provide more scalable deployment than hard upgrade?
>>
>
>
>
Re: Soft Upgrade and Downgrade
Posted by Kathey Marsden <km...@sbcglobal.net>.
Rick Hillegas wrote:
>
> Today, however, I was taught the following curious fact about
> DatabaseMetaData methods:
>
> o When the customer calls one of these methods, it causes the server
> to fault in some query text from the server jar files, to compile that
> query text, and to store the compiled form in the database. From that
> point onward, the compiled form is locked in place and determines the
> results of the DatabaseMetaData method for all subsequent callers.
>
> o The results from a DatabaseMetaData call therefore depend on whether
> the first user faulted in the text BEFORE or AFTER the Upgrade. It
> seems that this indeterminate behavior is induced by Soft Upgrade and
> maybe by Hard Upgrade as well.
>
>
>From what you describe it sounds like it might be DERBY-1107 which is a
general issue related to to any version change of Derby not just soft
upgrade, DERBY-1107 https://issues.apache.org/jira/browse/DERBY-1107.
There is also some history in that bug.
Historically the metadata queries have been trouble makers. I think the
metadata issue might be worthy of a Wiki page of its own and perhaps
some architectural thought put into how to best handle metadata queries
on version change and in general.
Kathey
Re: Soft Upgrade and Downgrade
Posted by Rick Hillegas <Ri...@Sun.COM>.
Mike Matrigali wrote:
> The way I look at it, derby has a very nice default. We call it soft
> upgrade, but in my mind it really is "no upgrade". We guarantee that
> if you do nothing to your existing app, then you can run it and the
> database against your current version of the derby software and all
> future versions of the derby software. I think this is a reasonable
> "automatic" default.
This was the conclusion I was happy to come to yesterday after Dan
answered my questions in the related thread titled "Hard vs. Soft
Upgrade": Soft Upgrade should be a misnomer, and "No Upgrade" should
describe the situation better.
Today, however, I was taught the following curious fact about
DatabaseMetaData methods:
o When the customer calls one of these methods, it causes the server to
fault in some query text from the server jar files, to compile that
query text, and to store the compiled form in the database. From that
point onward, the compiled form is locked in place and determines the
results of the DatabaseMetaData method for all subsequent callers.
o The results from a DatabaseMetaData call therefore depend on whether
the first user faulted in the text BEFORE or AFTER the Upgrade. It seems
that this indeterminate behavior is induced by Soft Upgrade and maybe by
Hard Upgrade as well.
Is this true? What other wormholes lurk in Soft Upgrade? I would think
that anyone who has to support Derby would be happier with deterministic
behavior.
>
> Because we provide no way to "go back" with database versions I believe
> it is reasonable to require the very slight user intervention of
> connecting and requiring the upgrade=true flag. If you want to use new
> stuff then you proactively tell us. This model is really important if
> we ever decide to do some sort of major data upgrade where the cost
> of hard upgrade may be hours depending on the size of the db and number
> of rows touched.
>
Upgrade is clearly an operation which stretches the "0-admin" claim a
bit past the cracking point.
Re: Soft Upgrade and Downgrade
Posted by Mike Matrigali <mi...@sbcglobal.net>.
The way I look at it, derby has a very nice default. We call it soft
upgrade, but in my mind it really is "no upgrade". We guarantee that
if you do nothing to your existing app, then you can run it and the
database against your current version of the derby software and all
future versions of the derby software. I think this is a reasonable
"automatic" default.
Because we provide no way to "go back" with database versions I believe
it is reasonable to require the very slight user intervention of
connecting and requiring the upgrade=true flag. If you want to use new
stuff then you proactively tell us. This model is really important if
we ever decide to do some sort of major data upgrade where the cost
of hard upgrade may be hours depending on the size of the db and number
of rows touched.
Oystein Grovlen - Sun Norway wrote:
> I agree that soft upgrade has a value, but I think you give soft upgrade
> credit for things that are actual not provided by soft upgrade. More
> comments below.
>
> Kathey Marsden wrote:
>
>> Rick Hillegas wrote:
>>
>>> 4) What is the customer problem solved by Soft Upgrade?
>>
>>
>>
>> Soft upgrade solves many problems for users , support and
>> development. It:
>>
>> - Minimizes end user intervention in accordance with Derby's charter
>> http://db.apache.org/derby/derby_charter.html.
>> which calls for ease of use and zero administration for end users.
>
>
> It is not soft upgrade that minimizes end user intervention. That is
> provided by automatic upgrade (soft or hard). In fact, soft upgrade
> increases user intervention since you will need to explicitly state that
> you want to do a hard upgrade. Without soft upgrade, hard upgrades
> could have been done automatically.
>
>> - Encourages upgrade and reduces the need to port fixes to old releases.
>
>
> I agree that people will probably be less hesistant to upgrade to new
> releases if they are able to revert to their old release. In my
> opinion, that is the only value of soft upgrade. In itself, I would not
> think that using the old data format should make it much less risky to
> upgrade than using a new data format. And if I had to downgrade due to
> problems with the new release, how do I know that this new faulty
> release has not corrupted my database?
>
>> - Provides applications, embedding or including Derby a scalable
>> deployment option for distributing new versions of Derby.
>
>
> How is soft upgrade provide more scalable deployment than hard upgrade?
>
Re: Soft Upgrade and Downgrade
Posted by Kathey Marsden <km...@sbcglobal.net>.
Øystein Grøvlen wrote:
>
> I think it is good to have soft upgrade as part of an upgrade path in
> order to be able to revert to the previous version in case of
> problems. I am bit more concerned about recommending users to run in
> soft upgrade more or less permanently since this mode is much less
> tested than a standard configuration.
>
I think often the users don't even realize they are running in soft
upgrade mode because for the embedded driver client and server are the
same thing. Folks tend to access their databases from different
applications using different versions of Derby. Take this scenario.
I am a developer and I tend to go back and forth accessing my database
between my IDE and App Server.
My IDE bundles Derby 10.1
My App Server bundles Derby 10.1
I decide to upgrade to the latest version of my IDE with Derby 10.3
My App Server doesn't yet officially support 10.3 so is still running at
10.1
- With soft upgrade it just works (nice).
- With no soft upgrade from 10.1 to 10.3 and requiring user
intervention, I realize I can't use my new IDE until I can upgrade my
database and App Server (not so nice).
- With automatic hard upgrade I find I can no longer use my database
with my App Server. (not very nice at all).
Similar situations happen if someone tries out a new gui database
browsers or some other tool on their database.
I agree that we should find ways to encourage users to move up their
data as well as their software. I see it very much like upgrading your
jdk. jdk 1.6 should read the old byte code. Derby should read the old
databases. You should always be able to plop down new jars, have things
work fine and be able to go back to the old version if you want, but as
you say, folks should be encouraged to move along, maybe a warning
would be good if you are more than one version behind.
Kathey
Re: Soft Upgrade and Downgrade
Posted by Daniel John Debrunner <dj...@apache.org>.
Øystein Grøvlen wrote:
> Kathey Marsden wrote:
>
>> First of all I want to say that there is a great dependence on soft
>> upgrade in the installed user base so if the community decided to
>> deprecate soft upgrade (which I am strongly against) it would need to be
>> just that a deprecation and some solution to the current usage models
>> would need to be found in advance. So this is not a 10.2 discussion.
>
>
> I am not suggesting that we should deprecate soft upgrade. I was just
> reacting to your claims that soft upgrade was providing a lot of
> wonderful stuff that is actually provided by automatic upgrade (soft or
> hard).
>
> I think it is good to have soft upgrade as part of an upgrade path in
> order to be able to revert to the previous version in case of problems.
> I am bit more concerned about recommending users to run in soft upgrade
> more or less permanently since this mode is much less tested than a
> standard configuration.
The design of the soft upgrade is meant to be low-risk. It's a mode
where certain operations are disabled.
Re: Soft Upgrade and Downgrade
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
Kathey Marsden wrote:
> First of all I want to say that there is a great dependence on soft
> upgrade in the installed user base so if the community decided to
> deprecate soft upgrade (which I am strongly against) it would need to be
> just that a deprecation and some solution to the current usage models
> would need to be found in advance. So this is not a 10.2 discussion.
I am not suggesting that we should deprecate soft upgrade. I was just
reacting to your claims that soft upgrade was providing a lot of
wonderful stuff that is actually provided by automatic upgrade (soft or
hard).
I think it is good to have soft upgrade as part of an upgrade path in
order to be able to revert to the previous version in case of problems.
I am bit more concerned about recommending users to run in soft
upgrade more or less permanently since this mode is much less tested
than a standard configuration.
--
Øystein
Re: Soft Upgrade and Downgrade
Posted by Kathey Marsden <km...@sbcglobal.net>.
Oystein Grovlen - Sun Norway wrote:
> I agree that soft upgrade has a value, but I think you give soft
> upgrade credit for things that are actual not provided by soft upgrade.
>
Just a few somewhat disjointed but relevant thoughts before I sign off
of this discussion for a while, because I need to fix some bugs.
First of all I want to say that there is a great dependence on soft
upgrade in the installed user base so if the community decided to
deprecate soft upgrade (which I am strongly against) it would need to be
just that a deprecation and some solution to the current usage models
would need to be found in advance. So this is not a 10.2 discussion.
But a few things to think about.
Many applications have their own soft upgrade model where they allow
you to downgrade if there was an issue with the upgrade. If we take
the example of the email client that is moving from Derby 10.0 to Derby
10.3, they cannot implement their own soft upgrade if we do not
provide that capability in Derby.
In the application server environment there is a a tangled web of Derby
versions. Many applications test and ship with one Derby version and
end up running in an application server with with another. We do not
yet support loading multiple derby versions in the same JVM, so the
applications are all tied together.
Sometimes there are scenarios where two JVM's will access the database
at different times, for instance an IDE running at one version accesses
the database and then it is used in the application server environment.
In terms of being able to revert, I am not so worried about corruption
by the new version. I am worried about application compatibility.
But like I said not a 10.2 discussion. Let's pick it back up next
release when there is less time crunch.
Kathey
Re: Soft Upgrade and Downgrade
Posted by Oystein Grovlen - Sun Norway <Oy...@Sun.COM>.
I agree that soft upgrade has a value, but I think you give soft upgrade
credit for things that are actual not provided by soft upgrade. More
comments below.
Kathey Marsden wrote:
> Rick Hillegas wrote:
>
>> 4) What is the customer problem solved by Soft Upgrade?
>
>
> Soft upgrade solves many problems for users , support and
> development. It:
>
> - Minimizes end user intervention in accordance with Derby's charter
> http://db.apache.org/derby/derby_charter.html.
> which calls for ease of use and zero administration for end users.
It is not soft upgrade that minimizes end user intervention. That is
provided by automatic upgrade (soft or hard). In fact, soft upgrade
increases user intervention since you will need to explicitly state that
you want to do a hard upgrade. Without soft upgrade, hard upgrades
could have been done automatically.
> - Encourages upgrade and reduces the need to port fixes to old releases.
I agree that people will probably be less hesistant to upgrade to new
releases if they are able to revert to their old release. In my
opinion, that is the only value of soft upgrade. In itself, I would not
think that using the old data format should make it much less risky to
upgrade than using a new data format. And if I had to downgrade due to
problems with the new release, how do I know that this new faulty
release has not corrupted my database?
> - Provides applications, embedding or including Derby a scalable
> deployment option for distributing new versions of Derby.
How is soft upgrade provide more scalable deployment than hard upgrade?
--
Øystein
Re: Soft Upgrade and Downgrade
Posted by Kathey Marsden <km...@sbcglobal.net>.
Rick Hillegas wrote:
>
> 4) What is the customer problem solved by Soft Upgrade?
Soft upgrade solves many problems for users , support and
development. It:
- Minimizes end user intervention in accordance with Derby's charter
http://db.apache.org/derby/derby_charter.html.
which calls for ease of use and zero administration for end users.
- Encourages upgrade and reduces the need to port fixes to old releases.
- Provides applications, embedding or including Derby a scalable
deployment option for distributing new versions of Derby.
Derby is normally embedded or included in applications and the jars get
distributed on cd or by download but the data does not. The databases
sit out at potentially millions of user sites, so it is important to
provide a seamless revertible upgrade. Keeping our super easy and
revertible upgrade at http://wiki.apache.org/db-derby/UpgradingTen
means a lot of work here but is worth it because whatever demands we put
on the end user or application developer can get multiplied by millions.
Some examples:
*Derby in an IDE or Application Server:*
In these environments the application being upgraded controls the Derby
version but doesn't know anything about the databases it is upgrading.
In the application server case they are also implicitly upgrading
applications they know nothing about. Sun puts Derby in J2EE and
NetBeans. Presumably Sun would like to ship 10.2 and in a later
version 10.3. Whatever upgrade implications we thrust upon the end user
are going to trickle down to every person that downloads J2EE or
NetBeans and then potentially to the users of whatever applications are
deployed on J2EE or developed with NetBeans.
*Derby in an embedded environment:*
Take the example of an email application. The makers of this software
want to make a new release with Derby 10.3. (Their last release was
with 10.1). They don't have any real changes to their application that
need the new functionality and know that this upgrade will impact the
Inbox's of hundreds of thousands of users. They have a requirement
that users be able to revert to the previous version if they choose
without restoring their data. They need soft upgrade of Derby or have
to stick with 10.1. (This example was inspired by IBM's current
effort to upgrade Lotus Notes for every one of IBM's 300,.000
employees. I am not actually sure how or if Derby soft upgrade fits
into the implementation of that).
*Derby in Support*
Currently IBM has 80 products and projects that use Cloudscape (Derby)
and include it with their distribution. I am not sure how many copies
of Derby that is at end user sites (I always think in the millions to
try to understand the scalability issues). We roll up fixes and
encourage our internal customers to upgrade early and often and tell
them that upgrading to the new jars should have no impact on their
application if they use the public interfaces. A super easy upgrade and
ensuring compatibility is key to receptiveness to rolled up fixes. If
we did not have this, they would demand one off patches and we would
have to port many patches to old versions.
There are many other examples too such as multiple components running in
the same jvm and sharing a derby version, but this email is getting too
long.
Kathey
Re: Soft Upgrade and Downgrade
Posted by Oystein Grovlen - Sun Norway <Oy...@Sun.COM>.
Kathey Marsden wrote:
> There is some info here:
> http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism
>
> I think we could safely add metadata to the list of items requiring
> upgrade thought.
This provides useful on how it is supposed to work, but is there any
guidelines on what a developer has to do in order to support this? If
this is not available, could someone provide pointers to where in the
code to look for examples on how this is done? Some aspects that comes
to mind is the code needed to:
- convert the data from the old format to the new format on hard upgrade.
- prevent functionality from being used in soft upgrade mode.
- use the old format in soft upgrade mode.
--
Øystein Grøvlen, Senior Staff Engineer
Sun Microsystems, Database Technology Group
Trondheim, Norway
Re: Soft Upgrade and Downgrade
Posted by Daniel John Debrunner <dj...@apache.org>.
Oystein Grovlen - Sun Norway wrote:
> Kathey Marsden wrote:
>
>> There is some info here:
>> http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism
>>
>
>
> I did not understand the purpose of item 7 in the list that describes
> the behavior when the engine version is different from the on-disk
> version. It says:
>
> 7. Requirement that EM <= 10 and DM >= 10.
This is a typo, should be greater than equal on EM, i.e.
7. Requirement that EM >= 10 and DM >= 10.
Also there is a typo on item 4, DM.DM should be replaced with DM.Dm.
Dan.
Re: Soft Upgrade and Downgrade
Posted by Oystein Grovlen - Sun Norway <Oy...@Sun.COM>.
Kathey Marsden wrote:
> There is some info here:
> http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism
I did not understand the purpose of item 7 in the list that describes
the behavior when the engine version is different from the on-disk
version. It says:
7. Requirement that EM <= 10 and DM >= 10.
Since EM < DM will cause the database not to boot (item 2), this seems
to say that in order to boot the database EM == DM == 10. In other
words, all future release needs to be called 10.x.
--
Øystein
Re: Soft Upgrade and Downgrade
Posted by Kathey Marsden <km...@sbcglobal.net>.
Rick Hillegas wrote:
> I'm looking for an explanation of how Soft Upgrade is supposed to
> work, from the customer's perspective and from the developer's
> perspective. I suppose that this also entails an explanation of the
> inverse operation, Soft Downgrade. I can't find this explanation.
>
There is some info here:
http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism
I think we could safely add metadata to the list of items requiring
upgrade thought.
Kathey