You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/04/27 20:11:15 UTC
[GitHub] [iceberg] yyanyy commented on pull request #2521: Core: fix NPE in manifests table for contains_nan column, update spec
yyanyy commented on pull request #2521:
URL: https://github.com/apache/iceberg/pull/2521#issuecomment-827897536
Thank you for the quick review and feedback everyone!
Regarding backward compatibility tests for reading manifest files, I think there are a few things worth mentioning:
1) Iceberg does have dedicated tests on verifying backwards compatibility for both [manifest file](https://github.com/apache/iceberg/blob/master/core/src/test/java/org/apache/iceberg/TestManifestWriterVersions.java) and [manifest list](https://github.com/apache/iceberg/blob/master/core/src/test/java/org/apache/iceberg/TestManifestListVersions.java).
2) even with these backward compatible tests, this issue can still occur (I did update the [test](https://github.com/apache/iceberg/blob/master/core/src/test/java/org/apache/iceberg/TestManifestListVersions.java#L221-L222) in the original PR). The reason is I think these are two separate issues:
- My understanding of the problem that's fixed by this specific PR is not related to generally reading of the manifest files/manifest list itself, but rather the display of a metadata table `manifests table` which is a separate logic.
- The problem before the fix was that the schema we specified for this manifest table is [wrong](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/ManifestsTable.java#L41) comparing to the correct specification we have for the actual requirement of manifest files [here](https://github.com/apache/iceberg/blob/master/api/src/main/java/org/apache/iceberg/ManifestFile.java#L58), and this mismatch caused the NPE problem since we require an originally nullable field to be not null.
- If we reused the schema construct we defined from `ManifestFile` (which is the source of truth in persisting manifest list file) in `ManifestsTable` we can avoid this problem completely, but currently `ManifestFile` doesn't want to expose them out, and has to specifically define the ID to ensure there's no duplication. I think we can still reuse them if we want, but that requires some code changes, and the current approach of duplicating definition of the same concepts in multiple places seems like a more widely adopted approach across Iceberg project.
- I think the ultimate way of preventing this specific problem is to create tests around metadata tables, and for this specific problem, we create v1 and v2 normal table, write data to them, and then query the manifest table and ensure output meets expected. However I think we don't have any test around metadata tables today.
I think the follow up item after this PR should be to create an issue to add metadata tables tests to make sure they don't break when reading different versions of tables. Any comment/feedback/suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org