You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/04/28 01:45:32 UTC
[GitHub] [iceberg] jackye1995 opened a new issue #2532: Add metadata tables tests to make sure they don't break when reading different versions of tables
jackye1995 opened a new issue #2532:
URL: https://github.com/apache/iceberg/issues/2532
based on discussion in #2521
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] itachi-sharingan commented on issue #2532: Add metadata tables tests to make sure they don't break when reading different versions of tables
Posted by GitBox <gi...@apache.org>.
itachi-sharingan commented on issue #2532:
URL: https://github.com/apache/iceberg/issues/2532#issuecomment-917640970
Is it ok to assume that this issue and #2542 are duplicate? I am asking this to completely understand the ask here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] taherk77 commented on issue #2532: Add metadata tables tests to make sure they don't break when reading different versions of tables
Posted by GitBox <gi...@apache.org>.
taherk77 commented on issue #2532:
URL: https://github.com/apache/iceberg/issues/2532#issuecomment-829237496
I would like to work on this issue. Can someone please guide a little and give more context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] itachi-sharingan commented on issue #2532: Add metadata tables tests to make sure they don't break when reading different versions of tables
Posted by GitBox <gi...@apache.org>.
itachi-sharingan commented on issue #2532:
URL: https://github.com/apache/iceberg/issues/2532#issuecomment-917650549
@RussellSpitzer @yyanyy
I am a newbie to this project so following question might be very trivial or wrong altogether, so I am ready to read through code and documentation to develop correct understanding of the project, but need some guidance to look at the correct place.
Question:-
Here https://github.com/apache/iceberg/pull/2521#issuecomment-828793923, what I understand is that to ensure backward compatibility for reading manifest files we should create couple of data files and manifest files explicitly in resources folder and create iceberg table from them, and let then run couple of test on the table created. Now can you point me to some code or test where manifest files are used to create an iceberg table, because I see that tables are getting created by catalog.buildTable which only takes table identifier and schema as input but not using data files and manifest files.
Also after figuring the above part, how can I find the fields that were introduced in different versions and remove them from manifest file in resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yyanyy commented on issue #2532: Add metadata tables tests to make sure they don't break when reading different versions of tables
Posted by GitBox <gi...@apache.org>.
yyanyy commented on issue #2532:
URL: https://github.com/apache/iceberg/issues/2532#issuecomment-918493847
@itachi-sharingan welcome to the community! Your understanding is correct. For creating iceberg table, the manifest/data files are not needed as the input since they should be eventually linked from the table metadata file, so such relationship should be preserved from the table metadata object and the manifest files in the resources folder.
For finding the files introduced, I think https://iceberg.apache.org/spec/#version-2 this could be a good place to check what's strictly required for v1 vs. v2; you can also create a v1 table (follow code path as in https://github.com/apache/iceberg/pull/2887 to find out the setting) and use the created files as starting point. I think we can have such setup for both v1 and v2 tables, so that in future when v3 is introduced we can continue to have this safety guard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ZhangchengHu0923 commented on issue #2532: Add metadata tables tests to make sure they don't break when reading different versions of tables
Posted by GitBox <gi...@apache.org>.
ZhangchengHu0923 commented on issue #2532:
URL: https://github.com/apache/iceberg/issues/2532#issuecomment-1084080574
I found this question was shelved for a long time,As it happens，I am a newbie to this projec,I'll pick up this work if u don't mind @itachi-sharingan @yyanyy @RussellSpitzer ,I am going to do the test by following the steps below, @yyanyy @RussellSpitzer could u please take a look to see weather my undertanding about the issue is correct,thx a lot.
step 1:create v1 v2 tables by TestTables
step 2:generate some data files and append to the tables
step 3:read diffrent kinds of meta info of both v1 and v2 table by subclasses of BaseMetadataTable，such as AllDataFilesTable/AllEntriesTable/HistoryTable, to see weather all theses subclasses work well
more questions:
> If my understanding is correct, what do u think is the good idea to divide the test work, divede according to v1/v2 version with 2 test class, or create one test class per kind of MetaDataTable?
> I plan to put the test class in iceberg-core/test,is that ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #2532: Add metadata tables tests to make sure they don't break when reading different versions of tables
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #2532:
URL: https://github.com/apache/iceberg/issues/2532#issuecomment-829275611
@taherk77 That would be awesome!
https://github.com/apache/iceberg/issues/2542#issue-870411640 Here are @yyanyy 's notes on the problem. Basically we want to make sure that whether you have a v1 table or a v2 table, the metadata table code reads correctly. Our previous tests cover converting manifests but don't make sure that manifests from both versions can be accessed by the same metadata table code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kyle-cx edited a comment on issue #2532: Add metadata tables tests to make sure they don't break when reading different versions of tables
Posted by GitBox <gi...@apache.org>.
kyle-cx edited a comment on issue #2532:
URL: https://github.com/apache/iceberg/issues/2532#issuecomment-1084080574
I found this question was shelved for a long time,As it happens，I am a newbie to this projec,I'll pick up this work if u don't mind @itachi-sharingan @yyanyy @RussellSpitzer ,I am going to do the test by following the steps below, @yyanyy @RussellSpitzer could u please take a look to see weather my undertanding about the issue is correct,thx a lot.
step 1:create v1 v2 tables by TestTables
step 2:generate some data files and append to the tables
step 3:read diffrent kinds of meta info of both v1 and v2 table by subclasses of BaseMetadataTable，such as AllDataFilesTable/AllEntriesTable/HistoryTable, to see weather all theses subclasses work well
one more questions:
> I plan to put the test class in iceberg-core/test,is that ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org