You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2019/12/24 09:39:02 UTC
[GitHub] [incubator-iceberg] chenjunjiedada opened a new pull request #718:
Update projection in addedFiles to support appending
chenjunjiedada opened a new pull request #718: Update projection in addedFiles to support appending
URL: https://github.com/apache/incubator-iceberg/pull/718
Sometimes we need to append data files from an existing table like below:
```java
Iterator<DataFile> datafiles = sourceTable.currentSnapshot().addedFiles().iterator();
AppendFiles appendFiles = targetTable.newAppend();
while (datafiles.hasNext()) {
appendFiles.appendFile(datafiles.next());
}
```
The `addedFiles` read from `Snapshot` has a projection while the schema from Avro writer doesn't have such projection and thus leads to error.
This PR enforces the manifest reader to do full columns projection to fix the problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#718: Update projection in addedFiles to support appending
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #718: Update projection in addedFiles to support appending
URL: https://github.com/apache/incubator-iceberg/pull/718#discussion_r361720401
##########
File path: core/src/test/java/org/apache/iceberg/TestFastAppend.java
##########
@@ -277,4 +278,27 @@ public void testRecoveryWithoutManifestList() {
Assert.assertTrue("Should commit the same new manifest",
metadata.currentSnapshot().manifests().contains(newManifest));
}
+
+ @Test
+ public void testAppendFilesFromTable() {
Review comment:
I think this should be in a test for `Snapshot`, not `FastAppend`. Can you start a new `TestSnapshot` suite?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] chenjunjiedada commented on a change in pull
request #718: Update projection in addedFiles to support appending
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on a change in pull request #718: Update projection in addedFiles to support appending
URL: https://github.com/apache/incubator-iceberg/pull/718#discussion_r361796053
##########
File path: core/src/test/java/org/apache/iceberg/TestFastAppend.java
##########
@@ -277,4 +278,27 @@ public void testRecoveryWithoutManifestList() {
Assert.assertTrue("Should commit the same new manifest",
metadata.currentSnapshot().manifests().contains(newManifest));
}
+
+ @Test
+ public void testAppendFilesFromTable() {
Review comment:
Sure, I will update it in the next commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#718: Update projection in addedFiles to support appending
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #718: Update projection in addedFiles to support appending
URL: https://github.com/apache/incubator-iceberg/pull/718#discussion_r361720568
##########
File path: core/src/test/java/org/apache/iceberg/TestFastAppend.java
##########
@@ -277,4 +278,27 @@ public void testRecoveryWithoutManifestList() {
Assert.assertTrue("Should commit the same new manifest",
metadata.currentSnapshot().manifests().contains(newManifest));
}
+
+ @Test
+ public void testAppendFilesFromTable() {
+ table.newFastAppend()
+ .appendFile(FILE_A)
+ .appendFile(FILE_B)
+ .commit();
+
+ // collect data files from deserialization
+ Iterator<DataFile> filesIterator = table.currentSnapshot().addedFiles().iterator();
+
+ table.newDelete().deleteFile(FILE_A).deleteFile(FILE_B).commit();
+
+ Snapshot oldSnapshot = table.currentSnapshot();
+
+ AppendFiles fastAppend = table.newFastAppend();
+ while (filesIterator.hasNext()) {
Review comment:
Why not use `for (DataFile file : table.currentSnapshot().addedFiles())`? Using a while loop and an explicit iterator seems odd.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] chenjunjiedada commented on a change in pull
request #718: Update projection in addedFiles to support appending
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on a change in pull request #718: Update projection in addedFiles to support appending
URL: https://github.com/apache/incubator-iceberg/pull/718#discussion_r361796231
##########
File path: core/src/test/java/org/apache/iceberg/TestFastAppend.java
##########
@@ -277,4 +278,27 @@ public void testRecoveryWithoutManifestList() {
Assert.assertTrue("Should commit the same new manifest",
metadata.currentSnapshot().manifests().contains(newManifest));
}
+
+ @Test
+ public void testAppendFilesFromTable() {
+ table.newFastAppend()
+ .appendFile(FILE_A)
+ .appendFile(FILE_B)
+ .commit();
+
+ // collect data files from deserialization
+ Iterator<DataFile> filesIterator = table.currentSnapshot().addedFiles().iterator();
+
+ table.newDelete().deleteFile(FILE_A).deleteFile(FILE_B).commit();
+
+ Snapshot oldSnapshot = table.currentSnapshot();
+
+ AppendFiles fastAppend = table.newFastAppend();
+ while (filesIterator.hasNext()) {
Review comment:
Make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue merged pull request #718: Update
projection in addedFiles to support appending
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #718: Update projection in addedFiles to support appending
URL: https://github.com/apache/incubator-iceberg/pull/718
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org