You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/04/20 22:15:09 UTC
[GitHub] [incubator-iceberg] rdblue opened a new pull request #941: Refactor metadata inheritance for manifest rewrites
rdblue opened a new pull request #941:
URL: https://github.com/apache/incubator-iceberg/pull/941
This refactors manifest rewrites done by `RewriteManifests`, `FastAppend`, and `MergeAppend`. Previously, the rewrite method was passed a `ManifestReader`, but now the rewrite method handles reader creation so that it can setup inherited metadata. For a rewritten manifest, the snapshot ID should always be present and not inherited because the records were already read and rewritten by the rewrite process. For an appended manifest, this adds `CopyMetadata` that sets the snapshot ID.
After this refactor, all metadata inheritance is setup in `ManifestFiles`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] aokolnychyi commented on issue #941: Refactor metadata inheritance for manifest rewrites
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on issue #941:
URL: https://github.com/apache/incubator-iceberg/pull/941#issuecomment-617435061
Let me check this one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #941: Refactor metadata inheritance for manifest rewrites
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #941:
URL: https://github.com/apache/incubator-iceberg/pull/941#discussion_r412529298
##########
File path: spark/src/test/java/org/apache/iceberg/spark/source/TestForwardCompatibility.java
##########
@@ -192,8 +197,7 @@ public void testSparkCanReadUnknownTransform() throws IOException {
manifestWriter.close();
}
- TableOperations ops = ((HasTableOperations) table).operations();
- ops.commit(ops.current(), ops.current().replaceCurrentSnapshot(new FakeSnapshot(manifestWriter.toManifestFile())));
+ table.newFastAppend().appendManifest(manifestWriter.toManifestFile()).commit();
Review comment:
This test needed to change because the update started catching that the manifest didn't have a snapshot ID for entries to inherit. The solution is to append the manifest like normal instead of using the commit API directly. To be able to commit the manifest without rewriting it (which fails on the fake partition data), this sets snapshot ID inheritance to true.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org