[jira] [Commented] (PHOENIX-6186) Store table metadata last modified timestamp in PTable / System.Catalog

["Geoffrey Jacoby (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17214299#comment-17214299 ] 

Geoffrey Jacoby commented on PHOENIX-6186:
------------------------------------------

[~kadir] suggested to me yesterday that the timestamp in PTable.getTimeStamp() could already function as this. After some digging, I don't think it can be used. PTable.getTimeStamp() doesn't come from a field in System.Catalog itself, but is created at PTable load time by taking the max of each Cell timestamp in a syscat row. While this would probably catch table creation, column addition, and column removal (because the column count will change on the header row), it would also catch changes to internal Phoenix properties like PHOENIX_TTL. 

The purpose of this JIRA is to identify the last time the metadata "shape" of a table, view, or index has changed in ways that would be relevant to an external application or change detection framework. Changing the existing behavior of PTable.getTimeStamp() to fit this invariant risks breaking other logic in Phoenix, such as the requirement that before a client can ALTER a table or view, its local copy of the metadata must be up to date. 

I believe, therefore, that this needs to be a new column (SCHEMA_TIMESTAMP? DDL_TIMESTAMP? Open to suggestions on name!), updated on CREATE, or an ALTER ADD/DROP COLUMN. The value would be the _server_ timestamp in MetadataEndpointImpl on the System.Catalog region that hosts the table or view parent row. 

I don't believe there needs to be any parent/child activity -- i.e, altering a view shouldn't affect the timestamp of its parent table or view, altering an index shouldn't affect the base table, etc. 

[~ChinmayKulkarni] [~yanxinyi] [~bharathv] [~kadir][~jainankit], fyi.

Wdyt?

> Store table metadata last modified timestamp in PTable / System.Catalog
> -----------------------------------------------------------------------
>
>                 Key: PHOENIX-6186
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-6186
>             Project: Phoenix
>          Issue Type: New Feature
>            Reporter: Geoffrey Jacoby
>            Assignee: Geoffrey Jacoby
>            Priority: Major
>             Fix For: 4.16.0
>
>
> There are many reasons why it's useful to know when a particular table's metadata was last modified. It's helpful when solving cache coherency problems, and also in order to interact with external schema registries which may have multiple versions of a particular schema and require a timestamp to resolve ambiguities. 
> This JIRA will add a last modified timestamp field to System.Catalog, to be updated both when creating a table/view and also when adding or removing a column. Changing purely internal Phoenix properties will not update the timestamp. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)