You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/01/28 09:26:08 UTC
[GitHub] [iceberg] melin opened a new issue #3997: Support Iceberg Metadata storage in a variety of engines
melin opened a new issue #3997:
URL: https://github.com/apache/iceberg/issues/3997
Most of Iceberg metadata is stored in the file system and is limited by NameNode performance. Storage engines such as RDBMS, Cassandra and mongodb can be supported through pluggable storage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on issue #3997: Support Iceberg Metadata storage in a variety of engines
Posted by GitBox <gi...@apache.org>.
flyrain commented on issue #3997:
URL: https://github.com/apache/iceberg/issues/3997#issuecomment-1028714442
With the new RestAPI design, we should be able to use RDBMS or key-value store to replace the metadata.json files. The new APIs is WIP, the client will be provided in Iceberg repo, but user needs to implement the server side once the APIs are ready. However, I believe the open source server will be there eventually, it is probably another project.
Other than metadata.json file, it needs a major overhaul to put manifest-list or manifest files into RDBMS/Key-value store. It is possible theoretically, but not sure it is the way people want to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] melin commented on issue #3997: Support Iceberg Metadata storage in a variety of engines
Posted by GitBox <gi...@apache.org>.
melin commented on issue #3997:
URL: https://github.com/apache/iceberg/issues/3997#issuecomment-1043129773
" file listing issue in Hudi metadata." => RFC - 15: HUDI Metadata Table and Cloud/DFS File Listing Improvements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on issue #3997: Support Iceberg Metadata storage in a variety of engines
Posted by GitBox <gi...@apache.org>.
flyrain commented on issue #3997:
URL: https://github.com/apache/iceberg/issues/3997#issuecomment-1042349608
Thanks for sharing. The Hudi metadata server makes sense generally. However, Iceberg doesn’t have the some of issues in Hudi, for example, file listing issue in Hudi metadata.
I list some benefits of an Iceberg metadata server.
1. Multiple-table transaction
2. Performance improvement
1. Avoid send the full metadata json file from client to server
2. Queuing multiple commits, other than each client resolves write-write conflict by themselves.
3. Caching resources like JDBC connections
3. Safer commits
1. Clients with different versions can safely commit, without worrying about overwriting the newer properties
2. Server side holds the truth of table format, upgrade to a newer version doesn’t need changes of all clients as long as API compatibility is kept.
There could be more benefits though.
References
* Related PRs: https://github.com/apache/iceberg/pull/3770
* Table metadata Rest API design Doc: https://docs.google.com/document/d/1D0R3G0slssEhggH5XnIzMwsUIP-c385Qp2sjv5E7e6E/edit#heading=h.eo4x0coo8esy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] melin commented on issue #3997: Support Iceberg Metadata storage in a variety of engines
Posted by GitBox <gi...@apache.org>.
melin commented on issue #3997:
URL: https://github.com/apache/iceberg/issues/3997#issuecomment-1029003852
Bytedance has implemented Hudi MetaStore Server，https://cwiki.apache.org/confluence/display/HUDI/RFC-36%3A+HUDI+Metastore+Server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org