You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gaborkaszab (via GitHub)" <gi...@apache.org> on 2024/08/23 15:30:19 UTC
[PR] Core: Remove one comment from FastAppend [iceberg]
gaborkaszab opened a new pull request, #10995:
URL: https://github.com/apache/iceberg/pull/10995
There is a comment for FastAppend that mentions the number of commit retries done by the class. I removed it for 2 reasons:
1) Not the FastAppend class does the commit so the commit retries shouldn't be commented there.
2) The number of commit retries is configurable via a table property so mentioning a hard-coded
value in the comment is incorrect.
Added a new test that adjusts the COMMIT_NUM_RETRIES table property. Adjusted existing tests to exercise the boundaries of the number of commit retries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #10995:
URL: https://github.com/apache/iceberg/pull/10995#discussion_r1816970746
##########
core/src/test/java/org/apache/iceberg/TestFastAppend.java:
##########
@@ -252,11 +252,36 @@ public void testFailure() {
assertThat(new File(newManifest.path())).doesNotExist();
}
+ @TestTemplate
+ public void testIncreaseNumRetries() {
+ TestTables.TestTableOperations ops = table.ops();
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
+
+ AppendFiles append = table.newFastAppend().appendFile(FILE_B);
+
+ // Default number of retries results in a failed commit
+ assertThatThrownBy(append::commit)
+ .isInstanceOf(CommitFailedException.class)
+ .hasMessage("Injected failure");
+
+ // After increasing the number of retries the commit succeeds
+ table
+ .updateProperties()
+ .set(
+ TableProperties.COMMIT_NUM_RETRIES,
+ String.valueOf(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1))
+ .commit();
+
+ append.commit();
+
+ validateSnapshot(null, readMetadata().currentSnapshot(), FILE_B);
+ }
+
@TestTemplate
public void testAppendManifestCleanup() throws IOException {
- // inject 5 failures
+ // inject failures
TestTables.TestTableOperations ops = table.ops();
- ops.failCommits(5);
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
Review Comment:
Makes sense. I reverted the changes in existing tests.
Thanks for taking a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10995:
URL: https://github.com/apache/iceberg/pull/10995#discussion_r1754002089
##########
core/src/main/java/org/apache/iceberg/FastAppend.java:
##########
@@ -24,20 +24,14 @@
import java.util.Set;
import org.apache.iceberg.encryption.EncryptedOutputFile;
import org.apache.iceberg.events.CreateSnapshotEvent;
-import org.apache.iceberg.exceptions.CommitFailedException;
import org.apache.iceberg.exceptions.RuntimeIOException;
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.util.CharSequenceSet;
-/**
- * {@link AppendFiles Append} implementation that adds a new manifest file for the write.
- *
- * <p>This implementation will attempt to commit 5 times before throwing {@link
Review Comment:
why not rather update the comment and refer to the table property that is being used for retries: `TableProperties#COMMIT_NUM_RETRIES_DEFAULT`?
In your PR description you mention that retries aren't done for `FastAppend`, but retries are done for all subclasses of `SnapshotProducer`.
Also if we update this javadoc, then we should align this for other subclasses of `SnapshotProducer` as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra merged PR #10995:
URL: https://github.com/apache/iceberg/pull/10995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #10995:
URL: https://github.com/apache/iceberg/pull/10995#discussion_r1768445973
##########
core/src/test/java/org/apache/iceberg/TestFastAppend.java:
##########
@@ -252,11 +252,36 @@ public void testFailure() {
assertThat(new File(newManifest.path())).doesNotExist();
}
+ @TestTemplate
+ public void testIncreaseNumRetries() {
+ TestTables.TestTableOperations ops = table.ops();
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
+
+ AppendFiles append = table.newFastAppend().appendFile(FILE_B);
+
+ // Default number of retries results in a failed commit
+ assertThatThrownBy(append::commit)
+ .isInstanceOf(CommitFailedException.class)
+ .hasMessage("Injected failure");
+
+ // After increasing the number of retries the commit succeeds
+ table
+ .updateProperties()
+ .set(
+ TableProperties.COMMIT_NUM_RETRIES,
+ String.valueOf(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1))
+ .commit();
+
+ append.commit();
+
+ validateSnapshot(null, readMetadata().currentSnapshot(), FILE_B);
+ }
+
@TestTemplate
public void testAppendManifestCleanup() throws IOException {
- // inject 5 failures
+ // inject failures
TestTables.TestTableOperations ops = table.ops();
- ops.failCommits(5);
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
Review Comment:
The value here is that we can test boundary values for the config and not just use a value that is within the range.
Also using the default value of the config instead of a hard-coded value is more future proof. One wouldn;t have to update tests once the default value changes for some reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #10995:
URL: https://github.com/apache/iceberg/pull/10995#discussion_r1768445973
##########
core/src/test/java/org/apache/iceberg/TestFastAppend.java:
##########
@@ -252,11 +252,36 @@ public void testFailure() {
assertThat(new File(newManifest.path())).doesNotExist();
}
+ @TestTemplate
+ public void testIncreaseNumRetries() {
+ TestTables.TestTableOperations ops = table.ops();
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
+
+ AppendFiles append = table.newFastAppend().appendFile(FILE_B);
+
+ // Default number of retries results in a failed commit
+ assertThatThrownBy(append::commit)
+ .isInstanceOf(CommitFailedException.class)
+ .hasMessage("Injected failure");
+
+ // After increasing the number of retries the commit succeeds
+ table
+ .updateProperties()
+ .set(
+ TableProperties.COMMIT_NUM_RETRIES,
+ String.valueOf(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1))
+ .commit();
+
+ append.commit();
+
+ validateSnapshot(null, readMetadata().currentSnapshot(), FILE_B);
+ }
+
@TestTemplate
public void testAppendManifestCleanup() throws IOException {
- // inject 5 failures
+ // inject failures
TestTables.TestTableOperations ops = table.ops();
- ops.failCommits(5);
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
Review Comment:
The value here is that we can test boundary values for the config and not just use a value that is within the range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10995:
URL: https://github.com/apache/iceberg/pull/10995#discussion_r1753988942
##########
core/src/test/java/org/apache/iceberg/TestFastAppend.java:
##########
@@ -252,11 +252,36 @@ public void testFailure() {
assertThat(new File(newManifest.path())).doesNotExist();
}
+ @TestTemplate
+ public void testIncreaseNumRetries() {
+ TestTables.TestTableOperations ops = table.ops();
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
+
+ AppendFiles append = table.newFastAppend().appendFile(FILE_B);
+
+ // Default number of retries results in a failed commit
+ assertThatThrownBy(append::commit)
+ .isInstanceOf(CommitFailedException.class)
+ .hasMessage("Injected failure");
+
+ // After increasing the number of retries the commit succeeds
+ table
+ .updateProperties()
+ .set(
+ TableProperties.COMMIT_NUM_RETRIES,
+ String.valueOf(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1))
+ .commit();
+
+ append.commit();
+
+ validateSnapshot(null, readMetadata().currentSnapshot(), FILE_B);
+ }
+
@TestTemplate
public void testAppendManifestCleanup() throws IOException {
- // inject 5 failures
+ // inject failures
TestTables.TestTableOperations ops = table.ops();
- ops.failCommits(5);
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
Review Comment:
TBH I don't see much value in updating this and the other tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10995:
URL: https://github.com/apache/iceberg/pull/10995#discussion_r1815163457
##########
core/src/test/java/org/apache/iceberg/TestFastAppend.java:
##########
@@ -252,11 +252,36 @@ public void testFailure() {
assertThat(new File(newManifest.path())).doesNotExist();
}
+ @TestTemplate
+ public void testIncreaseNumRetries() {
+ TestTables.TestTableOperations ops = table.ops();
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
+
+ AppendFiles append = table.newFastAppend().appendFile(FILE_B);
+
+ // Default number of retries results in a failed commit
+ assertThatThrownBy(append::commit)
+ .isInstanceOf(CommitFailedException.class)
+ .hasMessage("Injected failure");
+
+ // After increasing the number of retries the commit succeeds
+ table
+ .updateProperties()
+ .set(
+ TableProperties.COMMIT_NUM_RETRIES,
+ String.valueOf(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1))
+ .commit();
+
+ append.commit();
+
+ validateSnapshot(null, readMetadata().currentSnapshot(), FILE_B);
+ }
+
@TestTemplate
public void testAppendManifestCleanup() throws IOException {
- // inject 5 failures
+ // inject failures
TestTables.TestTableOperations ops = table.ops();
- ops.failCommits(5);
+ ops.failCommits(TableProperties.COMMIT_NUM_RETRIES_DEFAULT + 1);
Review Comment:
my main issue with these test updates is that those are not related to the scope of the PR itself, so I'd suggest to revert these changes. Adding a separate test method (which you did with `testIncreaseNumRetries()`) is fine but generally we try to avoid updating existing tests for no particular reason. Also I believe the goal of these tests was to use a very specific number of failures and such (independent from the default)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #10995:
URL: https://github.com/apache/iceberg/pull/10995#discussion_r1768454689
##########
core/src/main/java/org/apache/iceberg/FastAppend.java:
##########
@@ -24,20 +24,14 @@
import java.util.Set;
import org.apache.iceberg.encryption.EncryptedOutputFile;
import org.apache.iceberg.events.CreateSnapshotEvent;
-import org.apache.iceberg.exceptions.CommitFailedException;
import org.apache.iceberg.exceptions.RuntimeIOException;
import org.apache.iceberg.io.InputFile;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
import org.apache.iceberg.util.CharSequenceSet;
-/**
- * {@link AppendFiles Append} implementation that adds a new manifest file for the write.
- *
- * <p>This implementation will attempt to commit 5 times before throwing {@link
Review Comment:
Thanks for taking a look, @nastra!
Frankly, I don' really prefer redundant comments. I don't see why we'd want to add the same comment for all the subclasses of SnapshotProducer. First, as said in my description retries don't happen on that level of abstraction. Let's say for some reason the retry logic or the name of the config changes in the future. Then we have to remember that there is a comment about this in all the subclasses that we have to change now too.
What if I add a comment about the retries to SnapshotProducer itself? That's the level of abstraction where that logic happens, would be more suitable and we wouldn't have redundant comments in the subclasses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #10995:
URL: https://github.com/apache/iceberg/pull/10995#issuecomment-2435425999
Hi @nastra ,
Do you disagree with my reasoning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Remove one comment from FastAppend [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #10995:
URL: https://github.com/apache/iceberg/pull/10995#issuecomment-2441035715
Thanks for taking a look, @nastra !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org