You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Daniel John Debrunner <dj...@apache.org> on 2006/03/22 02:05:05 UTC
Re: Hard vs. Soft Upgrade
Discussion moved to derby-dev
Rick Hillegas wrote:
> Soft Upgrade is a feature which I think was introduced after I left
> Cloudscape. Please bear with me as I try to understand how Soft and Hard
> Upgrade interact.
>
> In the old days, when there was only Hard Upgrade, Upgrade satisfied the
> following contract:
>
> o Transitive - You could upgrade data from a lower version to any higher
> version by applying all of the intervening transforms in sequence.
> o Idempotent - Applying the same transform a second time was a nop.
>
> The version numbers themselves were organized in a tidy, ascending
> sequence. This determined the order in which the Upgrade transforms
> composed. So, for instance, you had versions h1, h2, h3, h4 with
> corresponding transforms H2, H3, H4 and you could expect that h4 = H4(
> H3( H2( h1 ) ) ).
>
> With Soft Upgrade, it appears that there is a second set of version
> numbers s1, s2, s3, s4 and a second set of corresponding transforms S2,
> S3, S4 and I would expect that s4 = S4( S3( S2( s1 ) ) ).
>
> But the situation might be more complicated than this. A database is now
> tagged with both a Hard and a Soft version number so that the database
> version is really an ordered pair ( hn, sm ) where m >= n. That is,
> version numbers are no longer a tidy, ascending sequence. Instead, the
> space of possible versions is a grid of points bounded by a triangular
> shape.
>
> What I am struggling to understand is how one upgrades from version (
> h1, s3 ) to ( h2, s? ). How does the H2 transform know what to do with
> the result of the S3 transform, given that S3 was coded after H2 shipped?
hm and sn are two indepdendent upgrade streams. Thus in your case you
would upgrading {h1, s3 ) to {h2, s3}. The upgrade code would see that
the soft upgrade is at a version greater than the engine is at (s2) and
so not apply any "safe" changes.
Soft upgrade typically will not make changes, we had one case prior to
open sourcing where upgrade could fix a bug in the system catalogs that
was causing a failure elsewhere. Any change by soft upgrade has to be a
fix that can be safely consumed by older versions of the engine, down to
the level of the database.
Currently there are no actions in soft upgrade, they are made in the
DD_version.applySoftChanges method which contains no actions. The
javadoc for that method tries to explain what it could do.
Dan.
Re: Hard vs. Soft Upgrade
Posted by Deepa Remesh <dr...@gmail.com>.
On 3/23/06, Daniel John Debrunner <dj...@apache.org> wrote:
> Deepa Remesh wrote:
>
> > On 3/22/06, Rick Hillegas <Ri...@sun.com> wrote:
> >
> >>It appears that changes to the database are partitioned into two
> >>buckets: those accomplished by Hard Upgrade and those accomplished by
> >>Soft Upgrade. Examples of Soft Upgrade changes appear under item (2)
> >>under the heading "Upgrading System Catalogs" on the webpage
> >>http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism.
> >>These are useful examples.
> >
> >
> > I was looking at this document and it is not very clear to me what can
> > be categorized as "safe changes" as mentioned in this statement:
> > "Apply "safe changes" from category 2) in a single transaction. An
> > example would be fixing incorrect information. ".
>
> Does this help (javadoc for DD_Version.applySafeChanges)?
>
> http://db.apache.org/derby/javadoc/engine/org/apache/derby/impl/sql/catalog/DD_Version.html#applySafeChanges(org.apache.derby.iapi.store.access.TransactionController,%20int,%20int)
>
Thanks.
> > I am looking at this
> > in the context of my work on the upgrade tests. Now, I am modifying
> > the master file for the test and I am not sure about one change done
> > as part of DERBY-475 (Add a system function mechanism and table of
> > functions, including a set of initial functions.). Current behaviour
> > is: the new functions are available after soft upgrade and downgrade
> > to old version after soft upgrade. Should these functions be available
> > after soft upgrade and downgrade ?
>
> Are you saying these new functions are available with 10.1 after 10.2
> has run against a 10.1 database?
>
> I would expect them to be not available when running 10.1, they are
> runtime functions provided by the 10.2 engine.
>
Sorry, I misread the master file. The current behaviour is: the new
functions are available after soft upgrade of 10.1 database but not
after downgrade. So I'll go ahead and modify the master file to
reflect this.
Thanks,
Deepa
Re: Hard vs. Soft Upgrade
Posted by Daniel John Debrunner <dj...@apache.org>.
Rick Hillegas wrote:
> Daniel John Debrunner wrote:
>
>> Deepa Remesh wrote:
>>
>>
>>
>>> On 3/22/06, Rick Hillegas <Ri...@sun.com> wrote:
>>>
>>>
>>>
>>>> It appears that changes to the database are partitioned into two
>>>> buckets: those accomplished by Hard Upgrade and those accomplished by
>>>> Soft Upgrade. Examples of Soft Upgrade changes appear under item (2)
>>>> under the heading "Upgrading System Catalogs" on the webpage
>>>> http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism.
>>>>
>>>> These are useful examples.
>>>>
>>>
>>> I was looking at this document and it is not very clear to me what can
>>> be categorized as "safe changes" as mentioned in this statement:
>>> "Apply "safe changes" from category 2) in a single transaction. An
>>> example would be fixing incorrect information. ".
>>
>>
>> Does this help (javadoc for DD_Version.applySafeChanges)?
>>
>> http://db.apache.org/derby/javadoc/engine/org/apache/derby/impl/sql/catalog/DD_Version.html#applySafeChanges(org.apache.derby.iapi.store.access.TransactionController,%20int,%20int)
>>
>>
>>
>>
> It's nice to see this documented, but it's not clearing up my confusion.
> The definition given here implies that a change is "safe" as long as you
> can reboot the database using old versions of Derby. What if, many hours
> after booting the old Derby version, a method returns odd results or the
> server actually crashes?
Sounds like a bug to me. Of course there's a risk that running *any* new
software could corrupt the database for the old engine, but the
intention is that soft upgrade allows one to revert versions.
> So far I have seen only two examples of "safe" changes
>
> o changing the nullability of a system column
Yes. The old engine can understand a column being nullable or not
nullable. This change is consumable by older versions.
> o changing the value in a set of rows to support new functionality (I'm
> a bit unclear on what this means, a concrete example would help)
That was not meant to be an example of a safe change.
The example for this (non-safe change) is say I implement some
functionality with two modes today and use the values 'A' and 'B' in a
system column to represent the two modes 'Orange' (A) and 'Bananna' (B).
Then in a different future release I add three new modes to the same
functionality, 'Apple', 'Pear' and 'Cherry'. I have two options:
1) leave the existing letters alone, and use Apple (R), Pear (P) and
Cherry (C), with Orange (A) and Bananna (B)
2) Decide that it would be less confusing if I used Apple (A), Pear (P)
and Cherry (C), with Orange (O) and Bananna (B). So I write upgrade code
that changes all the existing 'A's (Orange) to 'O'.
This type of change is not consumable by the older versions since they
are expecting the mode to be A or B, and have no code to handle O.
> It's hard for me to generalize from these two examples. Maybe the answer
> is:
>
> o We can't spell out the rules yet because we haven't played with this
> scheme long enough.
I think the rules are simple, I'm a little lost as to where the
confusion is coming in. Simply put, don't modify the database in such a
way that a previous version of the engine could not consume it.
> o Over several releases, we will collect more examples and distill a
> definition based on a case-by-case study of every upgrade-induced
> database change we introduce
Examples will help.
Dan.
Re: Hard vs. Soft Upgrade
Posted by Rick Hillegas <Ri...@Sun.COM>.
Daniel John Debrunner wrote:
>Deepa Remesh wrote:
>
>
>
>>On 3/22/06, Rick Hillegas <Ri...@sun.com> wrote:
>>
>>
>>
>>>It appears that changes to the database are partitioned into two
>>>buckets: those accomplished by Hard Upgrade and those accomplished by
>>>Soft Upgrade. Examples of Soft Upgrade changes appear under item (2)
>>>under the heading "Upgrading System Catalogs" on the webpage
>>>http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism.
>>>These are useful examples.
>>>
>>>
>>I was looking at this document and it is not very clear to me what can
>>be categorized as "safe changes" as mentioned in this statement:
>>"Apply "safe changes" from category 2) in a single transaction. An
>>example would be fixing incorrect information. ".
>>
>>
>
>Does this help (javadoc for DD_Version.applySafeChanges)?
>
>http://db.apache.org/derby/javadoc/engine/org/apache/derby/impl/sql/catalog/DD_Version.html#applySafeChanges(org.apache.derby.iapi.store.access.TransactionController,%20int,%20int)
>
>
>
It's nice to see this documented, but it's not clearing up my confusion.
The definition given here implies that a change is "safe" as long as you
can reboot the database using old versions of Derby. What if, many hours
after booting the old Derby version, a method returns odd results or the
server actually crashes?
So far I have seen only two examples of "safe" changes
o changing the nullability of a system column
o changing the value in a set of rows to support new functionality (I'm
a bit unclear on what this means, a concrete example would help)
It's hard for me to generalize from these two examples. Maybe the answer is:
o We can't spell out the rules yet because we haven't played with this
scheme long enough.
o Over several releases, we will collect more examples and distill a
definition based on a case-by-case study of every upgrade-induced
database change we introduce
Regards,
-Rick
Re: Hard vs. Soft Upgrade
Posted by Daniel John Debrunner <dj...@apache.org>.
Deepa Remesh wrote:
> On 3/22/06, Rick Hillegas <Ri...@sun.com> wrote:
>
>>It appears that changes to the database are partitioned into two
>>buckets: those accomplished by Hard Upgrade and those accomplished by
>>Soft Upgrade. Examples of Soft Upgrade changes appear under item (2)
>>under the heading "Upgrading System Catalogs" on the webpage
>>http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism.
>>These are useful examples.
>
>
> I was looking at this document and it is not very clear to me what can
> be categorized as "safe changes" as mentioned in this statement:
> "Apply "safe changes" from category 2) in a single transaction. An
> example would be fixing incorrect information. ".
Does this help (javadoc for DD_Version.applySafeChanges)?
http://db.apache.org/derby/javadoc/engine/org/apache/derby/impl/sql/catalog/DD_Version.html#applySafeChanges(org.apache.derby.iapi.store.access.TransactionController,%20int,%20int)
> I am looking at this
> in the context of my work on the upgrade tests. Now, I am modifying
> the master file for the test and I am not sure about one change done
> as part of DERBY-475 (Add a system function mechanism and table of
> functions, including a set of initial functions.). Current behaviour
> is: the new functions are available after soft upgrade and downgrade
> to old version after soft upgrade. Should these functions be available
> after soft upgrade and downgrade ?
Are you saying these new functions are available with 10.1 after 10.2
has run against a 10.1 database?
I would expect them to be not available when running 10.1, they are
runtime functions provided by the 10.2 engine.
Dan.
Re: Hard vs. Soft Upgrade
Posted by Deepa Remesh <dr...@gmail.com>.
On 3/22/06, Rick Hillegas <Ri...@sun.com> wrote:
>
> It appears that changes to the database are partitioned into two
> buckets: those accomplished by Hard Upgrade and those accomplished by
> Soft Upgrade. Examples of Soft Upgrade changes appear under item (2)
> under the heading "Upgrading System Catalogs" on the webpage
> http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism.
> These are useful examples.
I was looking at this document and it is not very clear to me what can
be categorized as "safe changes" as mentioned in this statement:
"Apply "safe changes" from category 2) in a single transaction. An
example would be fixing incorrect information. ". I am looking at this
in the context of my work on the upgrade tests. Now, I am modifying
the master file for the test and I am not sure about one change done
as part of DERBY-475 (Add a system function mechanism and table of
functions, including a set of initial functions.). Current behaviour
is: the new functions are available after soft upgrade and downgrade
to old version after soft upgrade. Should these functions be available
after soft upgrade and downgrade ?
I would appreciate if someone can explain a bit more about what is
categorized as "safe changes".
Thanks,
Deepa
Re: Hard vs. Soft Upgrade
Posted by Rick Hillegas <Ri...@Sun.COM>.
Thanks for the clarification, Dan. So it appears that Hard and Soft
Upgrade are two independent axes of change. Transforms along these axes
obey the following rules:
o Transitive
o Idempotent
o NOT Invertible
It appears that changes to the database are partitioned into two
buckets: those accomplished by Hard Upgrade and those accomplished by
Soft Upgrade. Examples of Soft Upgrade changes appear under item (2)
under the heading "Upgrading System Catalogs" on the webpage
http://db.apache.org/derby/papers/versionupgrade.html#Version+Upgrade+Mechanism.
These are useful examples. Do we have a process for guaranteeing that
future database changes end up in the right bucket?
Thanks,
-Rick
Daniel John Debrunner wrote:
>Discussion moved to derby-dev
>
>Rick Hillegas wrote:
>
>
>
>>Soft Upgrade is a feature which I think was introduced after I left
>>Cloudscape. Please bear with me as I try to understand how Soft and Hard
>>Upgrade interact.
>>
>>In the old days, when there was only Hard Upgrade, Upgrade satisfied the
>>following contract:
>>
>>o Transitive - You could upgrade data from a lower version to any higher
>>version by applying all of the intervening transforms in sequence.
>>o Idempotent - Applying the same transform a second time was a nop.
>>
>>The version numbers themselves were organized in a tidy, ascending
>>sequence. This determined the order in which the Upgrade transforms
>>composed. So, for instance, you had versions h1, h2, h3, h4 with
>>corresponding transforms H2, H3, H4 and you could expect that h4 = H4(
>>H3( H2( h1 ) ) ).
>>
>>With Soft Upgrade, it appears that there is a second set of version
>>numbers s1, s2, s3, s4 and a second set of corresponding transforms S2,
>>S3, S4 and I would expect that s4 = S4( S3( S2( s1 ) ) ).
>>
>>But the situation might be more complicated than this. A database is now
>>tagged with both a Hard and a Soft version number so that the database
>>version is really an ordered pair ( hn, sm ) where m >= n. That is,
>>version numbers are no longer a tidy, ascending sequence. Instead, the
>>space of possible versions is a grid of points bounded by a triangular
>>shape.
>>
>>What I am struggling to understand is how one upgrades from version (
>>h1, s3 ) to ( h2, s? ). How does the H2 transform know what to do with
>>the result of the S3 transform, given that S3 was coded after H2 shipped?
>>
>>
>
>hm and sn are two indepdendent upgrade streams. Thus in your case you
>would upgrading {h1, s3 ) to {h2, s3}. The upgrade code would see that
>the soft upgrade is at a version greater than the engine is at (s2) and
>so not apply any "safe" changes.
>
>Soft upgrade typically will not make changes, we had one case prior to
>open sourcing where upgrade could fix a bug in the system catalogs that
>was causing a failure elsewhere. Any change by soft upgrade has to be a
>fix that can be safely consumed by older versions of the engine, down to
>the level of the database.
>
>Currently there are no actions in soft upgrade, they are made in the
>DD_version.applySoftChanges method which contains no actions. The
>javadoc for that method tries to explain what it could do.
>
>Dan.
>
>
>