[jira] [Updated] (IGNITE-19230) Prune obsolete schema versions

["Roman Puchkovskiy (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IGNITE-19230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Roman Puchkovskiy updated IGNITE-19230:
---------------------------------------
    Description: 
Schema versions’ history might grow infinitely with time. We should prune the history we keep in memory by removing the schema versions SV which we will never need.

For table schemas, this means that no tuples exist in MVStorage such that the schema version would be used to read the tuple.

If we decide to bind the schema version to a tuple’s commitTs, then even one tuple written with a very old schema version (given that the tuple is still the newest version in its version chain) will prevent us from removing this schema version (and probably all schema versions following it).

On the other hand, storing a schema version ID with a tuple, we can have a background task that would upgrade tuples to the latest version of the schema, making it possible to later prune the old schema versions. We could maintain an LWM (lower watermark) per table that would be the least schema version that still can possibly have tuples written with it in the database, similarly to the MVStorage garbage collection design.

For index schemas, we might use the same LWM as for the tables owning them.

For schemas of other objects, this has to be defined.

> Prune obsolete schema versions
> ------------------------------
>
>                 Key: IGNITE-19230
>                 URL: https://issues.apache.org/jira/browse/IGNITE-19230
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Roman Puchkovskiy
>            Priority: Major
>              Labels: ignite-3
>
> Schema versions’ history might grow infinitely with time. We should prune the history we keep in memory by removing the schema versions SV which we will never need.
> For table schemas, this means that no tuples exist in MVStorage such that the schema version would be used to read the tuple.
> If we decide to bind the schema version to a tuple’s commitTs, then even one tuple written with a very old schema version (given that the tuple is still the newest version in its version chain) will prevent us from removing this schema version (and probably all schema versions following it).
> On the other hand, storing a schema version ID with a tuple, we can have a background task that would upgrade tuples to the latest version of the schema, making it possible to later prune the old schema versions. We could maintain an LWM (lower watermark) per table that would be the least schema version that still can possibly have tuples written with it in the database, similarly to the MVStorage garbage collection design.
> For index schemas, we might use the same LWM as for the tables owning them.
> For schemas of other objects, this has to be defined.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)