You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Mike Matrigali (JIRA)" <ji...@apache.org> on 2007/04/10 19:52:32 UTC
[jira] Created: (DERBY-2537) implement pushing collation info to
store, storing collation info in store metadata, and creating templates
based on store metadata
implement pushing collation info to store, storing collation info in store metadata, and creating templates based on store metadata
-----------------------------------------------------------------------------------------------------------------------------------
Key: DERBY-2537
URL: https://issues.apache.org/jira/browse/DERBY-2537
Project: Derby
Issue Type: Sub-task
Components: SQL, Store
Reporter: Mike Matrigali
Assigned To: Mike Matrigali
Fix For: 10.3.0.0
Implement subtasks associated with pushing and using collation info in store.
As was discussed on list the aproach will be:
o store collation info gets passed into createConglomerate() using an enhanced version of the exising CollumnOrdering interface
o store collation info will also have to get passed into interface that adds a column to a heap
o collation info will be added to the store metadata so that correct collation datatypes can be created internal to store where there
may no access to system catalogs (for instance at database recovery time). A new heap and btree metadata version will be
added which will included a "sparse" on disk format which will describe those columns that have a collation id that is different
than the default.
o soft upgrade will be handled by insuring old format is readable by existing code and automatically creates current version of in memory structures by
assuming all old format conglomerates have default collation info. Soft upgrade will never create new format heap or btree metadata.
o hard upgrade will write new format conglomerate metadata for conglomerates created after the upgrade, no work will be done at upgrade time.
Note the proposed implementation of collation feature in 10.3 will only allow new dbs to set alternate collation, so in 10.3 hard upgrade is not
actually necessary but provides a consistent approach to internal store structures. It also will allow for a future where more collation control
is allowed in dbs.
o store needs to create templates sometimes, to do this with correct collation info some of the Monitor functionality to create templates will be
moved from monitor into datatype factory and enhanced to take collaiton id info where it currently only takes format ids.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (DERBY-2537) implement pushing collation info to
store, storing collation info in store metadata, and creating templates
based on store metadata
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali resolved DERBY-2537.
-----------------------------------
Resolution: Fixed
> implement pushing collation info to store, storing collation info in store metadata, and creating templates based on store metadata
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-2537
> URL: https://issues.apache.org/jira/browse/DERBY-2537
> Project: Derby
> Issue Type: Sub-task
> Components: SQL, Store
> Reporter: Mike Matrigali
> Assignee: Mike Matrigali
> Fix For: 10.3.0.0
>
> Attachments: derby-2537_3.diff, derby2537_1.diff, derby2537_2.diff
>
>
> Implement subtasks associated with pushing and using collation info in store.
> As was discussed on list the aproach will be:
> o store collation info gets passed into createConglomerate() using an enhanced version of the exising CollumnOrdering interface
> o store collation info will also have to get passed into interface that adds a column to a heap
> o collation info will be added to the store metadata so that correct collation datatypes can be created internal to store where there
> may no access to system catalogs (for instance at database recovery time). A new heap and btree metadata version will be
> added which will included a "sparse" on disk format which will describe those columns that have a collation id that is different
> than the default.
> o soft upgrade will be handled by insuring old format is readable by existing code and automatically creates current version of in memory structures by
> assuming all old format conglomerates have default collation info. Soft upgrade will never create new format heap or btree metadata.
> o hard upgrade will write new format conglomerate metadata for conglomerates created after the upgrade, no work will be done at upgrade time.
> Note the proposed implementation of collation feature in 10.3 will only allow new dbs to set alternate collation, so in 10.3 hard upgrade is not
> actually necessary but provides a consistent approach to internal store structures. It also will allow for a future where more collation control
> is allowed in dbs.
> o store needs to create templates sometimes, to do this with correct collation info some of the Monitor functionality to create templates will be
> moved from monitor into datatype factory and enhanced to take collaiton id info where it currently only takes format ids.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2537) implement pushing collation info to
store, storing collation info in store metadata, and creating templates
based on store metadata
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali updated DERBY-2537:
----------------------------------
Attachment: derby-2537_3.diff
derby-2537_3.diff is not ready for checkin. The main goal of this checkin is to finish the work of creating
"correct" dvd templates in the store, using the store's collation metadata and the new interface that takes
a format id and collation id and gives back an template object which can be initialized by store with a
readExternal. The bulk of the changes are work to get the DataValueFactory into places where before we
called a static monitor routine. It passed all tests before being integrated with the latest change
to return DVD's from DataValueFactory.getNull(). After integration it failed as store needs
DataValueFactory.getNull() to support returning classes for RowLocations. The specific implementation
of Rowlocations is currently hidden from the data factory, but they are known to implement DVD.
I am not sure what the right way to implement this is. In this patch I added support for the heap row location
format id by calling the Monitor to get it from the format id. If this is not ok, does anyone have an alternate
suggestion?
I am rerunning all the tests to see if there is any other problems. If all the tests pass and I don't hear otherwise I will check this in, and if later a better suggestion on the rowlocation support comes I will be
happy to change it.
> implement pushing collation info to store, storing collation info in store metadata, and creating templates based on store metadata
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-2537
> URL: https://issues.apache.org/jira/browse/DERBY-2537
> Project: Derby
> Issue Type: Sub-task
> Components: SQL, Store
> Reporter: Mike Matrigali
> Assigned To: Mike Matrigali
> Fix For: 10.3.0.0
>
> Attachments: derby-2537_3.diff, derby2537_1.diff, derby2537_2.diff
>
>
> Implement subtasks associated with pushing and using collation info in store.
> As was discussed on list the aproach will be:
> o store collation info gets passed into createConglomerate() using an enhanced version of the exising CollumnOrdering interface
> o store collation info will also have to get passed into interface that adds a column to a heap
> o collation info will be added to the store metadata so that correct collation datatypes can be created internal to store where there
> may no access to system catalogs (for instance at database recovery time). A new heap and btree metadata version will be
> added which will included a "sparse" on disk format which will describe those columns that have a collation id that is different
> than the default.
> o soft upgrade will be handled by insuring old format is readable by existing code and automatically creates current version of in memory structures by
> assuming all old format conglomerates have default collation info. Soft upgrade will never create new format heap or btree metadata.
> o hard upgrade will write new format conglomerate metadata for conglomerates created after the upgrade, no work will be done at upgrade time.
> Note the proposed implementation of collation feature in 10.3 will only allow new dbs to set alternate collation, so in 10.3 hard upgrade is not
> actually necessary but provides a consistent approach to internal store structures. It also will allow for a future where more collation control
> is allowed in dbs.
> o store needs to create templates sometimes, to do this with correct collation info some of the Monitor functionality to create templates will be
> moved from monitor into datatype factory and enhanced to take collaiton id info where it currently only takes format ids.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2537) implement pushing collation info to
store, storing collation info in store metadata, and creating templates
based on store metadata
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali updated DERBY-2537:
----------------------------------
change 535770:
This checkin fixes create index, alter table, and bulk insert to correctly
set the collation in the persistent tables and indexes that they create.
> implement pushing collation info to store, storing collation info in store metadata, and creating templates based on store metadata
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-2537
> URL: https://issues.apache.org/jira/browse/DERBY-2537
> Project: Derby
> Issue Type: Sub-task
> Components: SQL, Store
> Reporter: Mike Matrigali
> Assigned To: Mike Matrigali
> Fix For: 10.3.0.0
>
> Attachments: derby-2537_3.diff, derby2537_1.diff, derby2537_2.diff
>
>
> Implement subtasks associated with pushing and using collation info in store.
> As was discussed on list the aproach will be:
> o store collation info gets passed into createConglomerate() using an enhanced version of the exising CollumnOrdering interface
> o store collation info will also have to get passed into interface that adds a column to a heap
> o collation info will be added to the store metadata so that correct collation datatypes can be created internal to store where there
> may no access to system catalogs (for instance at database recovery time). A new heap and btree metadata version will be
> added which will included a "sparse" on disk format which will describe those columns that have a collation id that is different
> than the default.
> o soft upgrade will be handled by insuring old format is readable by existing code and automatically creates current version of in memory structures by
> assuming all old format conglomerates have default collation info. Soft upgrade will never create new format heap or btree metadata.
> o hard upgrade will write new format conglomerate metadata for conglomerates created after the upgrade, no work will be done at upgrade time.
> Note the proposed implementation of collation feature in 10.3 will only allow new dbs to set alternate collation, so in 10.3 hard upgrade is not
> actually necessary but provides a consistent approach to internal store structures. It also will allow for a future where more collation control
> is allowed in dbs.
> o store needs to create templates sometimes, to do this with correct collation info some of the Monitor functionality to create templates will be
> moved from monitor into datatype factory and enhanced to take collaiton id info where it currently only takes format ids.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2537) implement pushing collation info to
store, storing collation info in store metadata, and creating templates
based on store metadata
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali updated DERBY-2537:
----------------------------------
Attachment: derby2537_2.diff
This patch is an updated version of the derby2537_2.diff patch. It adds a few missing new files and also includes changes to push the collation info from language down to store. Some paths still have default collation hardcoded and have been marked TODO-COLLATION for implementation in subsequent checkins. I plan to commit this patch. It has passed full nightly tests, except for those issues being reported in main trunk.
> implement pushing collation info to store, storing collation info in store metadata, and creating templates based on store metadata
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-2537
> URL: https://issues.apache.org/jira/browse/DERBY-2537
> Project: Derby
> Issue Type: Sub-task
> Components: SQL, Store
> Reporter: Mike Matrigali
> Assigned To: Mike Matrigali
> Fix For: 10.3.0.0
>
> Attachments: derby2537_1.diff, derby2537_2.diff
>
>
> Implement subtasks associated with pushing and using collation info in store.
> As was discussed on list the aproach will be:
> o store collation info gets passed into createConglomerate() using an enhanced version of the exising CollumnOrdering interface
> o store collation info will also have to get passed into interface that adds a column to a heap
> o collation info will be added to the store metadata so that correct collation datatypes can be created internal to store where there
> may no access to system catalogs (for instance at database recovery time). A new heap and btree metadata version will be
> added which will included a "sparse" on disk format which will describe those columns that have a collation id that is different
> than the default.
> o soft upgrade will be handled by insuring old format is readable by existing code and automatically creates current version of in memory structures by
> assuming all old format conglomerates have default collation info. Soft upgrade will never create new format heap or btree metadata.
> o hard upgrade will write new format conglomerate metadata for conglomerates created after the upgrade, no work will be done at upgrade time.
> Note the proposed implementation of collation feature in 10.3 will only allow new dbs to set alternate collation, so in 10.3 hard upgrade is not
> actually necessary but provides a consistent approach to internal store structures. It also will allow for a future where more collation control
> is allowed in dbs.
> o store needs to create templates sometimes, to do this with correct collation info some of the Monitor functionality to create templates will be
> moved from monitor into datatype factory and enhanced to take collaiton id info where it currently only takes format ids.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2537) implement pushing collation info to
store, storing collation info in store metadata, and creating templates
based on store metadata
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali updated DERBY-2537:
----------------------------------
I think I have updated all the code that creates base tables or indexes to pass correct collation ids, except for the 2 following
cases. Can anyone give me an idea how to get the collation types, basically access the DataTypeDescriptor of the columns
they are loading into a store table (see sections marked TODO-COLLATE):
The 2 classes left are: MaterializedResultSet.java and TemporaryRowHolderImpl.java
It may be that default collation is all right. The kind of thing I am worried about is one of these conglomerates be used to
create a hash table which then store would create a bunch of objects using default collation and then somewhere up the
tree a compare happening on the objects in the hash table.
> implement pushing collation info to store, storing collation info in store metadata, and creating templates based on store metadata
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-2537
> URL: https://issues.apache.org/jira/browse/DERBY-2537
> Project: Derby
> Issue Type: Sub-task
> Components: SQL, Store
> Reporter: Mike Matrigali
> Assigned To: Mike Matrigali
> Fix For: 10.3.0.0
>
> Attachments: derby-2537_3.diff, derby2537_1.diff, derby2537_2.diff
>
>
> Implement subtasks associated with pushing and using collation info in store.
> As was discussed on list the aproach will be:
> o store collation info gets passed into createConglomerate() using an enhanced version of the exising CollumnOrdering interface
> o store collation info will also have to get passed into interface that adds a column to a heap
> o collation info will be added to the store metadata so that correct collation datatypes can be created internal to store where there
> may no access to system catalogs (for instance at database recovery time). A new heap and btree metadata version will be
> added which will included a "sparse" on disk format which will describe those columns that have a collation id that is different
> than the default.
> o soft upgrade will be handled by insuring old format is readable by existing code and automatically creates current version of in memory structures by
> assuming all old format conglomerates have default collation info. Soft upgrade will never create new format heap or btree metadata.
> o hard upgrade will write new format conglomerate metadata for conglomerates created after the upgrade, no work will be done at upgrade time.
> Note the proposed implementation of collation feature in 10.3 will only allow new dbs to set alternate collation, so in 10.3 hard upgrade is not
> actually necessary but provides a consistent approach to internal store structures. It also will allow for a future where more collation control
> is allowed in dbs.
> o store needs to create templates sometimes, to do this with correct collation info some of the Monitor functionality to create templates will be
> moved from monitor into datatype factory and enhanced to take collaiton id info where it currently only takes format ids.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-2537) implement pushing collation info to
store, storing collation info in store metadata, and creating templates
based on store metadata
Posted by "Knut Anders Hatlen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499027 ]
Knut Anders Hatlen commented on DERBY-2537:
-------------------------------------------
Are there any plans to investigate what caused the performance regression Olav reported in DERBY-2657?
> implement pushing collation info to store, storing collation info in store metadata, and creating templates based on store metadata
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-2537
> URL: https://issues.apache.org/jira/browse/DERBY-2537
> Project: Derby
> Issue Type: Sub-task
> Components: SQL, Store
> Reporter: Mike Matrigali
> Assigned To: Mike Matrigali
> Fix For: 10.3.0.0
>
> Attachments: derby-2537_3.diff, derby2537_1.diff, derby2537_2.diff
>
>
> Implement subtasks associated with pushing and using collation info in store.
> As was discussed on list the aproach will be:
> o store collation info gets passed into createConglomerate() using an enhanced version of the exising CollumnOrdering interface
> o store collation info will also have to get passed into interface that adds a column to a heap
> o collation info will be added to the store metadata so that correct collation datatypes can be created internal to store where there
> may no access to system catalogs (for instance at database recovery time). A new heap and btree metadata version will be
> added which will included a "sparse" on disk format which will describe those columns that have a collation id that is different
> than the default.
> o soft upgrade will be handled by insuring old format is readable by existing code and automatically creates current version of in memory structures by
> assuming all old format conglomerates have default collation info. Soft upgrade will never create new format heap or btree metadata.
> o hard upgrade will write new format conglomerate metadata for conglomerates created after the upgrade, no work will be done at upgrade time.
> Note the proposed implementation of collation feature in 10.3 will only allow new dbs to set alternate collation, so in 10.3 hard upgrade is not
> actually necessary but provides a consistent approach to internal store structures. It also will allow for a future where more collation control
> is allowed in dbs.
> o store needs to create templates sometimes, to do this with correct collation info some of the Monitor functionality to create templates will be
> moved from monitor into datatype factory and enhanced to take collaiton id info where it currently only takes format ids.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (DERBY-2537) implement pushing collation info to
store, storing collation info in store metadata, and creating templates
based on store metadata
Posted by "Mike Matrigali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DERBY-2537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Matrigali updated DERBY-2537:
----------------------------------
Attachment: derby2537_1.diff
derby2537_1.diff provides the initial framework for reading/writing the new collaiton metadata in store. I ran all the tests and got a few errors that I am still looking at - I think these are not related to my changes but will resolve before committing the change. This change is an incremental step, it just stores the default collate info - more work is needed to get the new collate info down to store.
Note when this gets checked in you will get errors accessing any existing databases created off the trunk. We don't handle upgrade within versions in the trunk. Upgrade from 10.2 db's will be fully supported when released.
> implement pushing collation info to store, storing collation info in store metadata, and creating templates based on store metadata
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: DERBY-2537
> URL: https://issues.apache.org/jira/browse/DERBY-2537
> Project: Derby
> Issue Type: Sub-task
> Components: SQL, Store
> Reporter: Mike Matrigali
> Assigned To: Mike Matrigali
> Fix For: 10.3.0.0
>
> Attachments: derby2537_1.diff
>
>
> Implement subtasks associated with pushing and using collation info in store.
> As was discussed on list the aproach will be:
> o store collation info gets passed into createConglomerate() using an enhanced version of the exising CollumnOrdering interface
> o store collation info will also have to get passed into interface that adds a column to a heap
> o collation info will be added to the store metadata so that correct collation datatypes can be created internal to store where there
> may no access to system catalogs (for instance at database recovery time). A new heap and btree metadata version will be
> added which will included a "sparse" on disk format which will describe those columns that have a collation id that is different
> than the default.
> o soft upgrade will be handled by insuring old format is readable by existing code and automatically creates current version of in memory structures by
> assuming all old format conglomerates have default collation info. Soft upgrade will never create new format heap or btree metadata.
> o hard upgrade will write new format conglomerate metadata for conglomerates created after the upgrade, no work will be done at upgrade time.
> Note the proposed implementation of collation feature in 10.3 will only allow new dbs to set alternate collation, so in 10.3 hard upgrade is not
> actually necessary but provides a consistent approach to internal store structures. It also will allow for a future where more collation control
> is allowed in dbs.
> o store needs to create templates sometimes, to do this with correct collation info some of the Monitor functionality to create templates will be
> moved from monitor into datatype factory and enhanced to take collaiton id info where it currently only takes format ids.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.