You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/07 03:40:22 UTC
[GitHub] [iceberg] lintingbin2009 opened a new issue, #5219: The metadata file is too large
lintingbin2009 opened a new issue, #5219:
URL: https://github.com/apache/iceberg/issues/5219
We have a table with thousands of columns. Every time a column is added, all the old columns will be retained in the schemas field of the metadata, and will not be deleted again. In this way, after hundreds of column addition operations, the metadata will have a size of several hundred MB.
Is there any way to clean up the schemas field. Does the metadata need to save all the historical schemas if it is just adding columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #5219: The metadata file is too large
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #5219:
URL: https://github.com/apache/iceberg/issues/5219#issuecomment-1178457621
Yes as I noted, this was something we used internally and I didn't push for it to be merged here because I wasn't sure there was any utility. This also would not cover your specific issue but provides a template for what could be a solution. As for community interest, I'm sure we could get something like this in if it is not solely for fixing our internal issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] closed issue #5219: The metadata file is too large
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #5219: The metadata file is too large
URL: https://github.com/apache/iceberg/issues/5219
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lintingbin2009 commented on issue #5219: The metadata file is too large
Posted by GitBox <gi...@apache.org>.
lintingbin2009 commented on issue #5219:
URL: https://github.com/apache/iceberg/issues/5219#issuecomment-1178454729
@RussellSpitzer This sounds like a good idea. However, the iceberg community staff did not express any of their opinions, so your code has not been merged into master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #5219: The metadata file is too large
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #5219:
URL: https://github.com/apache/iceberg/issues/5219#issuecomment-1178417191
I actually wrote something for cleaning up Partition Specs because of a slightly related issue.
https://github.com/apache/iceberg/pull/3462
But we could always extend this to schema's as well. Basically just a "cleanMetadata" function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #5219: The metadata file is too large
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5219:
URL: https://github.com/apache/iceberg/issues/5219#issuecomment-1396265308
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] github-actions[bot] commented on issue #5219: The metadata file is too large
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5219:
URL: https://github.com/apache/iceberg/issues/5219#issuecomment-1371568331
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org