You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "istreeter (via GitHub)" <gi...@apache.org> on 2025/02/15 20:12:31 UTC
[I] Print un-pretty metadata JSON files without whitespace [iceberg]
istreeter opened a new issue, #12281:
URL: https://github.com/apache/iceberg/issues/12281
### Feature Request / Improvement
Currently, metadata files are pretty-printed, with lots of new-lines and whitespace indentations. [This is the relevant line of code](https://github.com/apache/iceberg/blob/abb47830e7df7dc2ae93c74b0ad97f06cdd37aad/core/src/main/java/org/apache/iceberg/TableMetadataParser.java#L131), which uses the Jackson default pretty printer.
If we could write metadata files without redundant whitespace, then it would save some storage space, and network overhead.
This will have have most impact for tables with large metadata files. For example, I have seen a metadata files which was 53.6MB. After removing whitespace, this was reduced to 41.4MB. I have read other issues in github which mention gigabyte-scale metadata files, e.g. in #9734.
I cannot think of any downside of this suggested change. Metadata files are mainly read by machines not humans. And if a human does want to inspect a metadata file, then it is fairly easy to prettify a JSON file when needed.
I'd be happy to open a PR for this, if you think it's a good idea? It seems like an easy way to get a small but noticeable performance improvement for reads and writes.
### Query engine
None
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Print un-pretty metadata JSON files without whitespace [iceberg]
Posted by "istreeter (via GitHub)" <gi...@apache.org>.
istreeter commented on issue #12281:
URL: https://github.com/apache/iceberg/issues/12281#issuecomment-2663400749
> does removing the pretty causes any issues for rendering in common ui tools or any other reader ?
It's hard to guess which UI tools get used to read Iceberg metadata files.
If someone opened it in a very primitive text editor, then yes it would look ugly. But I'm fairly sure most modern developer tools have a way to format json for viewing.
> would be best to take this up in dev-list as well
Good idea -- I will join the dev list and ask the question over there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Print un-pretty metadata JSON files without whitespace [iceberg]
Posted by "istreeter (via GitHub)" <gi...@apache.org>.
istreeter commented on issue #12281:
URL: https://github.com/apache/iceberg/issues/12281#issuecomment-2663509614
In terms of just reducing file size, then yes you're right either compression or binary format has more impact. But that sounds like a difficult breaking change. It could be difficult to roll out, and even more difficult to roll back from.
Whereas stripping whitespace seems like a much easier win. If we assume parsers are indifferent to whitespace, then this is completely non-breaking.
By the way, I just sent a mail to the dev-list about this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Print un-pretty metadata JSON files without whitespace [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on issue #12281:
URL: https://github.com/apache/iceberg/issues/12281#issuecomment-2663451777
I'm also curious if supporting a binary format would make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Print un-pretty metadata JSON files without whitespace [iceberg]
Posted by "bryanck (via GitHub)" <gi...@apache.org>.
bryanck commented on issue #12281:
URL: https://github.com/apache/iceberg/issues/12281#issuecomment-2663447560
Turning on metadata file compression is the best option to reduce size, though stripping out formatting did yield ~5% savings in one large file I tested even after compressing. It's easy enough to run it through `jq` if you want it formatted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Print un-pretty metadata JSON files without whitespace [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on issue #12281:
URL: https://github.com/apache/iceberg/issues/12281#issuecomment-2661555646
This is interesting, does removing the pretty causes any issues for rendering in common ui tools or any other reader ?
> I have seen a metadata files which was 53.6MB. After removing whitespace, this was reduced to 41.4M
~22% reduction in the sounds awesome ! Looks like having pretty existed from day 1 in the codebase. would be best to take this up in dev-list as well
cc @bryanck, you might be interested ^^^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Print un-pretty metadata JSON files without whitespace [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra closed issue #12281: Print un-pretty metadata JSON files without whitespace
URL: https://github.com/apache/iceberg/issues/12281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org