upgrade and compiled statements in SYSSTATEMENTS - question

[Mike Matrigali <mi...@sbcglobal.net>]

My assumption for both soft and hard upgrade on changes to
objects in compiled parts of queries was that I did not have
to worry about upgrade.  That any change to query plan data
structures stored on disk would be taken care of automatically
by just dropping them on any release number change.

Is that valid?

Re: upgrade and compiled statements in SYSSTATEMENTS - question

[Mike Matrigali <mi...@sbcglobal.net>]

Daniel John Debrunner wrote:
> Knut Anders Hatlen wrote:
> 
>> Mike Matrigali <mi...@sbcglobal.net> writes:
>>
>>> Knut.Hatlen@Sun.COM wrote:
>>>
>>>> Mike Matrigali <mi...@sbcglobal.net> writes:
>>>>
>>>>
>>>>> My assumption for both soft and hard upgrade on changes to
>>>>> objects in compiled parts of queries was that I did not have
>>>>> to worry about upgrade.  That any change to query plan data
>>>>> structures stored on disk would be taken care of automatically
>>>>> by just dropping them on any release number change.
>>>>>
>>>>> Is that valid?
>>>>
>>>>
>>>> If you by "any release number change" mean change of major or minor
>>>> release number, yes. On soft upgrade, the queries will be read directly
>>>> from org/apache/derby/impl/jdbc/metadata.properties or
>>>> org/apache/derby/impl/sql/catalog/metadata_net.properties. On hard
>>>> upgrade, the statements in SYSSTATEMENTS will be dropped, re-read from
>>>> metadata(_net).properties and recompiled.
>>>>
>>>> There are some more details on the wiki:
>>>> http://wiki.apache.org/db-derby/MetadataUpgrade
>>>
>>> Thanks for the pointer.  From the wiki it seems like we don't drop
>>> SYSSTATEMENTS for soft upgrade.
>>
>>
>> That's correct. We don't drop them since we want to be able to go back
>> to the old version without needing any downgrade code.
>>
>>> It also sounds like we prevent reads
>>> from SYSSTATEMENTS for soft upgrade.
>>
>>
>> Yes, since the queries might have been changed between the versions we
>> cannot use the old ones in SYSSTATEMENTS.
>>
>>> Is there meant to be something
>>> preventing writes to SYSSTATEMENTS for soft upgrade (ie. is there
>>> anything preventing recompile from updating SYSSTATEMENTS)?
>>
>>
>> To my knowledge, there is nothing that prevents writes to SYSSTATEMENTS
>> (except the mechanism that prevents metadata calls from using it), but I
>> thought SYSSTATEMENTS was only written to during database creation and
>> hard upgrade. And recompiles shouldn't happen as long as the metadata
>> calls don't use the stored statements, right? There could of course be
>> code (even user code) that executes the statements directly, but I'm not
>> aware of any part of the Derby code doing that.
>>
> 
> SYSSTATEMENTS is also used for the compiled plans for the action 
> statements for triggers, thus it can be written to at any time.
> 
> Any version number change causes all plans in SYSSTATEMENTS to become 
> invalid and require an automatic re-compile.
In terms of what this means for store, I think this means that the
system is expecting to be able to issue an update of a column where
the current value of the column is "old format" and the new value
of the column is "new format" - but there is no current code that 
understands how to properly read the "old format" in.  Is this the
requirement?  There is no way the current store interfaces commit
to provide this ability, as a side effect it may mostly work.


In general I know
that store can not support this, it is particularly a problem for
btree's. For instance during abort of such
an update store may need to instantiate an instance of both the old
and new columns to compare them.  This definitely happens during
logical undo in btree's.

In specific, it may be that the only need is to support these columns
in heaps, not btrees.
> 
> Dan.
> 
> 
> 

Re: upgrade and compiled statements in SYSSTATEMENTS - question

[Daniel John Debrunner <dj...@apache.org>]

Knut Anders Hatlen wrote:
> Mike Matrigali <mi...@sbcglobal.net> writes:
> 
>> Knut.Hatlen@Sun.COM wrote:
>>> Mike Matrigali <mi...@sbcglobal.net> writes:
>>>
>>>
>>>> My assumption for both soft and hard upgrade on changes to
>>>> objects in compiled parts of queries was that I did not have
>>>> to worry about upgrade.  That any change to query plan data
>>>> structures stored on disk would be taken care of automatically
>>>> by just dropping them on any release number change.
>>>>
>>>> Is that valid?
>>>
>>> If you by "any release number change" mean change of major or minor
>>> release number, yes. On soft upgrade, the queries will be read directly
>>> from org/apache/derby/impl/jdbc/metadata.properties or
>>> org/apache/derby/impl/sql/catalog/metadata_net.properties. On hard
>>> upgrade, the statements in SYSSTATEMENTS will be dropped, re-read from
>>> metadata(_net).properties and recompiled.
>>>
>>> There are some more details on the wiki:
>>> http://wiki.apache.org/db-derby/MetadataUpgrade
>> Thanks for the pointer.  From the wiki it seems like we don't drop
>> SYSSTATEMENTS for soft upgrade.
> 
> That's correct. We don't drop them since we want to be able to go back
> to the old version without needing any downgrade code.
> 
>> It also sounds like we prevent reads
>> from SYSSTATEMENTS for soft upgrade.
> 
> Yes, since the queries might have been changed between the versions we
> cannot use the old ones in SYSSTATEMENTS.
> 
>> Is there meant to be something
>> preventing writes to SYSSTATEMENTS for soft upgrade (ie. is there
>> anything preventing recompile from updating SYSSTATEMENTS)?
> 
> To my knowledge, there is nothing that prevents writes to SYSSTATEMENTS
> (except the mechanism that prevents metadata calls from using it), but I
> thought SYSSTATEMENTS was only written to during database creation and
> hard upgrade. And recompiles shouldn't happen as long as the metadata
> calls don't use the stored statements, right? There could of course be
> code (even user code) that executes the statements directly, but I'm not
> aware of any part of the Derby code doing that.
> 

SYSSTATEMENTS is also used for the compiled plans for the action 
statements for triggers, thus it can be written to at any time.

Any version number change causes all plans in SYSSTATEMENTS to become 
invalid and require an automatic re-compile.

Dan.

Re: upgrade and compiled statements in SYSSTATEMENTS - question

[Knut Anders Hatlen <Kn...@Sun.COM>]

Mike Matrigali <mi...@sbcglobal.net> writes:

> Knut.Hatlen@Sun.COM wrote:
>> Mike Matrigali <mi...@sbcglobal.net> writes:
>>
>>
>>>My assumption for both soft and hard upgrade on changes to
>>>objects in compiled parts of queries was that I did not have
>>>to worry about upgrade.  That any change to query plan data
>>>structures stored on disk would be taken care of automatically
>>>by just dropping them on any release number change.
>>>
>>>Is that valid?
>>
>>
>> If you by "any release number change" mean change of major or minor
>> release number, yes. On soft upgrade, the queries will be read directly
>> from org/apache/derby/impl/jdbc/metadata.properties or
>> org/apache/derby/impl/sql/catalog/metadata_net.properties. On hard
>> upgrade, the statements in SYSSTATEMENTS will be dropped, re-read from
>> metadata(_net).properties and recompiled.
>>
>> There are some more details on the wiki:
>> http://wiki.apache.org/db-derby/MetadataUpgrade
>
> Thanks for the pointer.  From the wiki it seems like we don't drop
> SYSSTATEMENTS for soft upgrade.

That's correct. We don't drop them since we want to be able to go back
to the old version without needing any downgrade code.

> It also sounds like we prevent reads
> from SYSSTATEMENTS for soft upgrade.

Yes, since the queries might have been changed between the versions we
cannot use the old ones in SYSSTATEMENTS.

> Is there meant to be something
> preventing writes to SYSSTATEMENTS for soft upgrade (ie. is there
> anything preventing recompile from updating SYSSTATEMENTS)?

To my knowledge, there is nothing that prevents writes to SYSSTATEMENTS
(except the mechanism that prevents metadata calls from using it), but I
thought SYSSTATEMENTS was only written to during database creation and
hard upgrade. And recompiles shouldn't happen as long as the metadata
calls don't use the stored statements, right? There could of course be
code (even user code) that executes the statements directly, but I'm not
aware of any part of the Derby code doing that.

-- 
Knut Anders

Re: upgrade and compiled statements in SYSSTATEMENTS - question

[Mike Matrigali <mi...@sbcglobal.net>]

Knut.Hatlen@Sun.COM wrote:
> Mike Matrigali <mi...@sbcglobal.net> writes:
> 
> 
>>My assumption for both soft and hard upgrade on changes to
>>objects in compiled parts of queries was that I did not have
>>to worry about upgrade.  That any change to query plan data
>>structures stored on disk would be taken care of automatically
>>by just dropping them on any release number change.
>>
>>Is that valid?
> 
> 
> If you by "any release number change" mean change of major or minor
> release number, yes. On soft upgrade, the queries will be read directly
> from org/apache/derby/impl/jdbc/metadata.properties or
> org/apache/derby/impl/sql/catalog/metadata_net.properties. On hard
> upgrade, the statements in SYSSTATEMENTS will be dropped, re-read from
> metadata(_net).properties and recompiled.
> 
> There are some more details on the wiki:
> http://wiki.apache.org/db-derby/MetadataUpgrade

Thanks for the pointer.  From the wiki it seems like we don't drop
SYSSTATEMENTS for soft upgrade.  It also sounds like we prevent reads
from SYSSTATEMENTS for soft upgrade.  Is there meant to be something
preventing writes to SYSSTATEMENTS for soft upgrade (ie. is there
anything preventing recompile from updating SYSSTATEMENTS)?

> 

Re: upgrade and compiled statements in SYSSTATEMENTS - question

[Kn...@Sun.COM]

Mike Matrigali <mi...@sbcglobal.net> writes:

> My assumption for both soft and hard upgrade on changes to
> objects in compiled parts of queries was that I did not have
> to worry about upgrade.  That any change to query plan data
> structures stored on disk would be taken care of automatically
> by just dropping them on any release number change.
>
> Is that valid?

If you by "any release number change" mean change of major or minor
release number, yes. On soft upgrade, the queries will be read directly
from org/apache/derby/impl/jdbc/metadata.properties or
org/apache/derby/impl/sql/catalog/metadata_net.properties. On hard
upgrade, the statements in SYSSTATEMENTS will be dropped, re-read from
metadata(_net).properties and recompiled.

There are some more details on the wiki:
http://wiki.apache.org/db-derby/MetadataUpgrade

-- 
Knut Anders