You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/01 17:03:56 UTC
[GitHub] [iceberg] Yanam opened a new pull request #3641: API: add methods that operate on meta files
Yanam opened a new pull request #3641:
URL: https://github.com/apache/iceberg/pull/3641
The purpose of this PR is to split the io() methods that operate on data files and meta files.
The reason is that when I implemented the custom catalog, I found that the API can only customize the read/write logic of meta.json files, but had no control over the manifest files that also belong to metadata. Since the data file and manifest file are basically operated by the FileIO object obtained by [Table.io() ](https://github.com/apache/iceberg/blob/master/api/src/main/java/org/apache/iceberg/Table.java)or [TableOperations.io() ](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/TableOperations.java)method, the read/write logic of the two files cannot be split. Separating the FileIO of the meta file may be more conducive to custom catalog.
Of course there is still a lot of code involved to fully implement the splitting of data files and manifest files, so if this solution is feasible, there is still a lot of work to follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Yanam closed pull request #3641: API: add methods that operate on meta files
Posted by GitBox <gi...@apache.org>.
Yanam closed pull request #3641:
URL: https://github.com/apache/iceberg/pull/3641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Yanam commented on pull request #3641: API: add methods that operate on meta files
Posted by GitBox <gi...@apache.org>.
Yanam commented on pull request #3641:
URL: https://github.com/apache/iceberg/pull/3641#issuecomment-1001904464
> Can you elaborate more on your use case?
>
> This would be a large refactor, and we tend to prefer making the FileIO composable (see `ResolvingFileIO`).
>
> It would help if you could elaborate why such a large proposed change would be beneficial. Possibly there's an existing way to handle what you need - or one that would require significantly less change within Iceberg.
My way of dealing with it is not very good. This one you provided will solve my problem. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3641: API: add methods that operate on meta files
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3641:
URL: https://github.com/apache/iceberg/pull/3641#issuecomment-986407957
Can you elaborate more on your use case?
This would be a large refactor, and we tend to prefer making the FileIO composable (see `ResolvingFileIO`).
It would help if you could elaborate why such a large proposed change would be beneficial. Possibly there's an existing way to handle what you need - or one that would require significantly less change within Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org