Re: [I] Improve read times and reduce size of metadata.json by storing schemas in external files [iceberg]

["moneymoroch (via GitHub)" <gi...@apache.org>]

moneymoroch commented on issue #9734:
URL: https://github.com/apache/iceberg/issues/9734#issuecomment-2037811832

   @bdilday Definitely interested in this feature. 
   
   We had an issue where metadata.json grew so large with schema changes (in the GBs) that it was causing performance issues within the query engine we were using (Trino). For this particular table, we only retained a few days of snapshot history and ended up manually pruning schema versions that were no used by recent snapshots to cut down on file size. 
   
   However, the solution you proposed would be more robust and suitable for cases where you want to time travel on very wide tables 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org