[GitHub] [iceberg] yyanyy commented on pull request #2096: Core: add schema id and schemas to table metadata

[GitBox <gi...@apache.org>]

yyanyy commented on pull request #2096:
URL: https://github.com/apache/iceberg/pull/2096#issuecomment-771320010


   > > Do we want to add a last-assigned-schema-id to table metadata?
   > 
   > My initial thought was no because the schema IDs are completely local to a metadata file. We can reassign all IDs from one file to the next and as long as they are internally consistent we are okay. The IDs are not embedded in separate files that might get improperly used.
   > 
   > But, the Nessie comment you pointed to is a good one to think about. I think it is still okay because Nessie merges would either be fast-forward, or would re-apply changes and create entirely new metadata files. So the internal consistency argument still holds.
   > 
   > > Do we want to assign special schema IDs to metadata tables, to avoid potential collision on schema IDs?
   > 
   > I don't think so.
   > 
   > > Do we want to add snapshot-id to only history entries, or Snapshot interface?
   > 
   > I would add it to Snapshot, not just History. Snapshot is what will be used for time travel queries. I would avoid adding too much metadata to history.
   > 
   > > I think currently when replacing a table, earlier history entries/snapshotLog will be reset to empty (second to last argument in here). Is this expected? do we want to fix this as a separate issue?
   > 
   > I think we should follow up and keep the history. I think the reason for this is because we previously didn't have any compatibility across schemas (IDs were completely reassigned, so time travel would be incorrect). But that's fixed now.
   
   Thank you for the review and the input! I'll create a separate issue to mention the history problem and link here. Do you have any suggestion on if we want to add schema id to the history table? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org