You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ChristinaTech (via GitHub)" <gi...@apache.org> on 2023/03/25 01:36:28 UTC
[GitHub] [iceberg] ChristinaTech opened a new pull request, #7198: AWS: Check commit status after failed commit if AWS client performed retries
ChristinaTech opened a new pull request, #7198:
URL: https://github.com/apache/iceberg/pull/7198
This PR fixes the issue described by #7151 where a commit can succeed but Glue or Dynamo backed tables see a failure and delete the now-committed metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1151208227
##########
aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbTableOperations.java:
##########
@@ -105,6 +106,8 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
boolean newTable = base == null;
String newMetadataLocation = writeNewMetadataIfRequired(newTable, metadata);
CommitStatus commitStatus = CommitStatus.FAILURE;
+ AwsClientFactories.InternalRetryDetector retryDetector =
Review Comment:
I don't think we can use it as an instance variable? Since the boolean flag will not be reset, I think we need to create an new instance every execution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1157372844
##########
aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbTableOperations.java:
##########
@@ -116,20 +118,30 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
.build());
checkMetadataLocation(table, base);
Map<String, String> properties = prepareProperties(table, newMetadataLocation);
- persistTable(tableKey, table, properties);
+ persistTable(tableKey, table, properties, retryDetector);
commitStatus = CommitStatus.SUCCESS;
- } catch (ConditionalCheckFailedException e) {
- throw new CommitFailedException(
- e, "Cannot commit %s: concurrent update detected", tableName());
} catch (CommitFailedException e) {
// any explicit commit failures are passed up and out to the retry handler
throw e;
} catch (RuntimeException persistFailure) {
- LOG.error(
- "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
- fullTableName,
- persistFailure);
- commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ boolean conditionCheckFailed = persistFailure instanceof ConditionalCheckFailedException;
+
+ // If we got an exception we weren't expecting, or we got a ConditionalCheckFailedException
+ // but retries were performed, attempt to reconcile the actual commit status.
+ if (!conditionCheckFailed || retryDetector.retried()) {
+ LOG.error(
+ "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
+ fullTableName,
+ persistFailure);
Review Comment:
This message predates this PR, and I generally try to avoid making changes beyond the scope of what I am trying to fix. That said, I don't actually take issue with the change in log level or error message save one thing. I am going to modify your new message slightly so it still reports the table name like the current message does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#issuecomment-1502322048
Thanks for the work @ChristinaTech ! Thanks for the reviews @amogh-jahagirdar @JonasJ-ap !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1151304377
##########
aws/src/integration/java/org/apache/iceberg/aws/glue/GlueTestBase.java:
##########
@@ -83,13 +82,7 @@ public static void beforeClass() {
AwsProperties properties = new AwsProperties();
properties.setS3FileIoDeleteBatchSize(10);
glueCatalog.initialize(
- catalogName,
- testBucketPath,
- properties,
- glue,
- LockManagers.defaultLockManager(),
- fileIO,
- ImmutableMap.of());
+ catalogName, testBucketPath, properties, glue, null, fileIO, ImmutableMap.of());
Review Comment:
I was and still am on the fence about removing the `InMemoryLockManager` being used here, but considering it hides the existence of Table-corrupting bugs from the integ tests, leaned towards doing so. I would like to get other opinions on whether this is the right call, and if not, how we will make sure that this kind of issue has enough testing coverage going forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 merged pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 merged PR #7198:
URL: https://github.com/apache/iceberg/pull/7198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1151193853
##########
aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java:
##########
@@ -88,6 +91,32 @@ private static AwsClientFactory loadClientFactory(String impl, Map<String, Strin
return factory;
}
+ /**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. Not for public use.
+ */
+ public static class InternalRetryDetector implements MetricPublisher {
Review Comment:
actually given the fact that this is referred across class, seems like it should not be nested in AwsClientFactories? I get now why it's public. In that case we should name it something more public, maybe just `RetryDetector` is good enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1153794991
##########
aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogCommitFailure.java:
##########
@@ -123,6 +128,74 @@ public void testConcurrentModificationExceptionDoesNotCheckCommitStatus() {
Assert.assertEquals("No new metadata files should exist", 2, metadataFileCount(ops.current()));
}
+ @Test
+ public void testCheckCommitStatusAfterRetries() {
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ TableMetadata metadataV1 = spyOps.current();
+ simulateRetriedCommit(spyOps, true);
+ updateTable(table, spyOps);
+
+ Assert.assertNotEquals("Current metadata should have changed", metadataV1, spyOps.current());
+ Assert.assertTrue("Current metadata should still exist", metadataFileExists(spyOps.current()));
+ Assert.assertEquals(
+ "No new metadata files should exist", 2, metadataFileCount(spyOps.current()));
+ }
+
+ @Test
+ public void testNoRetryAwarenessCorruptsTable() {
+ // This test exists to replicate the issue the prior test validates the fix for
Review Comment:
I shall update the comment to provide a link to it, assuming the conversation that follows this does not end up deciding to remove this test entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1151225170
##########
aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java:
##########
@@ -88,6 +91,32 @@ private static AwsClientFactory loadClientFactory(String impl, Map<String, Strin
return factory;
}
+ /**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. Not for public use.
+ */
+ public static class InternalRetryDetector implements MetricPublisher {
Review Comment:
Will make it to its own utility class and rename it to `RetryDetector`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "JonasJ-ap (via GitHub)" <gi...@apache.org>.
JonasJ-ap commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1153579660
##########
aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogCommitFailure.java:
##########
@@ -123,6 +128,74 @@ public void testConcurrentModificationExceptionDoesNotCheckCommitStatus() {
Assert.assertEquals("No new metadata files should exist", 2, metadataFileCount(ops.current()));
}
+ @Test
+ public void testCheckCommitStatusAfterRetries() {
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ TableMetadata metadataV1 = spyOps.current();
+ simulateRetriedCommit(spyOps, true);
+ updateTable(table, spyOps);
+
+ Assert.assertNotEquals("Current metadata should have changed", metadataV1, spyOps.current());
+ Assert.assertTrue("Current metadata should still exist", metadataFileExists(spyOps.current()));
+ Assert.assertEquals(
+ "No new metadata files should exist", 2, metadataFileCount(spyOps.current()));
+ }
+
+ @Test
+ public void testNoRetryAwarenessCorruptsTable() {
+ // This test exists to replicate the issue the prior test validates the fix for
Review Comment:
Shall we put the link to the issue (https://github.com/apache/iceberg/issues/7151) here? I think it can help others understand the purpose of these tests better?
##########
aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogCommitFailure.java:
##########
@@ -123,6 +128,74 @@ public void testConcurrentModificationExceptionDoesNotCheckCommitStatus() {
Assert.assertEquals("No new metadata files should exist", 2, metadataFileCount(ops.current()));
}
+ @Test
+ public void testCheckCommitStatusAfterRetries() {
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ TableMetadata metadataV1 = spyOps.current();
+ simulateRetriedCommit(spyOps, true);
+ updateTable(table, spyOps);
+
+ Assert.assertNotEquals("Current metadata should have changed", metadataV1, spyOps.current());
+ Assert.assertTrue("Current metadata should still exist", metadataFileExists(spyOps.current()));
+ Assert.assertEquals(
+ "No new metadata files should exist", 2, metadataFileCount(spyOps.current()));
+ }
+
+ @Test
+ public void testNoRetryAwarenessCorruptsTable() {
+ // This test exists to replicate the issue the prior test validates the fix for
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ simulateRetriedCommit(spyOps, false);
Review Comment:
[Question] I was wondering if there are any real-world cases where GlueTableOps may receive a ConcurrentModificationException but the table is still successfully committed without any retries. Or is this simulated failure case solely serving as a control group for the fix implemented in this PR?
It seems a little redundant to have an integration test solely for replicating the issue that has already been addressed by this PR. Could you please clarify this for me? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1156739798
##########
aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbTableOperations.java:
##########
@@ -116,20 +118,30 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
.build());
checkMetadataLocation(table, base);
Map<String, String> properties = prepareProperties(table, newMetadataLocation);
- persistTable(tableKey, table, properties);
+ persistTable(tableKey, table, properties, retryDetector);
commitStatus = CommitStatus.SUCCESS;
- } catch (ConditionalCheckFailedException e) {
- throw new CommitFailedException(
- e, "Cannot commit %s: concurrent update detected", tableName());
} catch (CommitFailedException e) {
// any explicit commit failures are passed up and out to the retry handler
throw e;
} catch (RuntimeException persistFailure) {
- LOG.error(
- "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
- fullTableName,
- persistFailure);
- commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ boolean conditionCheckFailed = persistFailure instanceof ConditionalCheckFailedException;
+
+ // If we got an exception we weren't expecting, or we got a ConditionalCheckFailedException
+ // but retries were performed, attempt to reconcile the actual commit status.
+ if (!conditionCheckFailed || retryDetector.retried()) {
+ LOG.error(
+ "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
+ fullTableName,
+ persistFailure);
Review Comment:
Should this be at a `warn` level instead? If the commit did end up being successful, then it may mislead people if it's at error level. also nit on the message, maybe "Received unexpected failure when committing, validating if commit ended up succeeding"
##########
aws/src/main/java/org/apache/iceberg/aws/util/RetryDetector.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.util;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import software.amazon.awssdk.core.metrics.CoreMetric;
+import software.amazon.awssdk.metrics.MetricCollection;
+import software.amazon.awssdk.metrics.MetricPublisher;
+
+/**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. This class can be attached to an AWS API call and checked after to determine if retries
+ * occurred.
+ */
+public class RetryDetector implements MetricPublisher {
Review Comment:
Could we avoid making this public for now? Since it's used within the `TableOperations` implementation we should be able to make it package private. At a later point, if there's a desire to expose this outside of the AWS module we can make it public
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java:
##########
@@ -154,47 +156,29 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
Table glueTable = getGlueTable();
checkMetadataLocation(glueTable, base);
Map<String, String> properties = prepareProperties(glueTable, newMetadataLocation);
- persistGlueTable(glueTable, properties, metadata);
+ persistGlueTable(glueTable, properties, metadata, retryDetector);
commitStatus = CommitStatus.SUCCESS;
} catch (CommitFailedException e) {
throw e;
- } catch (ConcurrentModificationException e) {
- throw new CommitFailedException(
- e, "Cannot commit %s because Glue detected concurrent update", tableName());
- } catch (software.amazon.awssdk.services.glue.model.AlreadyExistsException e) {
- throw new AlreadyExistsException(
- e,
- "Cannot commit %s because its Glue table already exists when trying to create one",
- tableName());
- } catch (EntityNotFoundException e) {
- throw new NotFoundException(
- e, "Cannot commit %s because Glue cannot find the requested entity", tableName());
- } catch (AccessDeniedException e) {
- throw new ForbiddenException(
- e, "Cannot commit %s because Glue cannot access the requested resources", tableName());
- } catch (software.amazon.awssdk.services.glue.model.ValidationException e) {
- throw new ValidationException(
- e,
- "Cannot commit %s because Glue encountered a validation exception "
- + "while accessing requested resources",
- tableName());
} catch (RuntimeException persistFailure) {
- LOG.error(
- "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
- fullTableName,
- persistFailure);
+ boolean isAwsServiceException = persistFailure instanceof AwsServiceException;
- if (persistFailure instanceof AwsServiceException) {
- int statusCode = ((AwsServiceException) persistFailure).statusCode();
- if (statusCode >= 500 && statusCode < 600) {
- commitStatus = CommitStatus.FAILURE;
- } else {
- throw persistFailure;
- }
- } else {
+ // If we got an exception we weren't expecting, or we got an AWS service exception
+ // but retries were performed, attempt to reconcile the actual commit status.
+ if (!isAwsServiceException || retryDetector.retried()) {
+ LOG.error(
Review Comment:
Same as above on warn level and message
##########
aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbTableOperations.java:
##########
@@ -116,20 +118,30 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
.build());
checkMetadataLocation(table, base);
Map<String, String> properties = prepareProperties(table, newMetadataLocation);
- persistTable(tableKey, table, properties);
+ persistTable(tableKey, table, properties, retryDetector);
commitStatus = CommitStatus.SUCCESS;
- } catch (ConditionalCheckFailedException e) {
- throw new CommitFailedException(
- e, "Cannot commit %s: concurrent update detected", tableName());
} catch (CommitFailedException e) {
// any explicit commit failures are passed up and out to the retry handler
throw e;
} catch (RuntimeException persistFailure) {
- LOG.error(
- "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
- fullTableName,
- persistFailure);
- commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ boolean conditionCheckFailed = persistFailure instanceof ConditionalCheckFailedException;
+
+ // If we got an exception we weren't expecting, or we got a ConditionalCheckFailedException
+ // but retries were performed, attempt to reconcile the actual commit status.
+ if (!conditionCheckFailed || retryDetector.retried()) {
+ LOG.error(
+ "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
+ fullTableName,
+ persistFailure);
+ commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ }
+
+ // If we got ConditionalCheckFailedException, but find we
+ // succeeded on a retry that threw an exception, skip this exception.
+ if (commitStatus != CommitStatus.SUCCESS && conditionCheckFailed) {
+ throw new CommitFailedException(
+ persistFailure, "Cannot commit %s: concurrent update detected", tableName());
+ }
Review Comment:
Shouldn't we still throw if the commit status is unknown?
##########
aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbTableOperations.java:
##########
@@ -116,20 +118,30 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
.build());
checkMetadataLocation(table, base);
Map<String, String> properties = prepareProperties(table, newMetadataLocation);
- persistTable(tableKey, table, properties);
+ persistTable(tableKey, table, properties, retryDetector);
commitStatus = CommitStatus.SUCCESS;
- } catch (ConditionalCheckFailedException e) {
- throw new CommitFailedException(
- e, "Cannot commit %s: concurrent update detected", tableName());
} catch (CommitFailedException e) {
// any explicit commit failures are passed up and out to the retry handler
throw e;
} catch (RuntimeException persistFailure) {
- LOG.error(
- "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
- fullTableName,
- persistFailure);
- commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ boolean conditionCheckFailed = persistFailure instanceof ConditionalCheckFailedException;
+
+ // If we got an exception we weren't expecting, or we got a ConditionalCheckFailedException
+ // but retries were performed, attempt to reconcile the actual commit status.
+ if (!conditionCheckFailed || retryDetector.retried()) {
+ LOG.error(
+ "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
+ fullTableName,
+ persistFailure);
+ commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ }
+
+ // If we got ConditionalCheckFailedException, but find we
+ // succeeded on a retry that threw an exception, skip this exception.
Review Comment:
Nit: I get what the inline comment is going for and it's true, but I would probably just leave this out since it doesn't directly correspond with the block below. Or a comment could be something like "Only throw after checking the commit status with consideration to retries, and if the condition check failed".
##########
aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbTableOperations.java:
##########
@@ -116,20 +118,30 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
.build());
checkMetadataLocation(table, base);
Map<String, String> properties = prepareProperties(table, newMetadataLocation);
- persistTable(tableKey, table, properties);
+ persistTable(tableKey, table, properties, retryDetector);
commitStatus = CommitStatus.SUCCESS;
- } catch (ConditionalCheckFailedException e) {
- throw new CommitFailedException(
- e, "Cannot commit %s: concurrent update detected", tableName());
} catch (CommitFailedException e) {
// any explicit commit failures are passed up and out to the retry handler
throw e;
} catch (RuntimeException persistFailure) {
- LOG.error(
- "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
- fullTableName,
- persistFailure);
- commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ boolean conditionCheckFailed = persistFailure instanceof ConditionalCheckFailedException;
+
+ // If we got an exception we weren't expecting, or we got a ConditionalCheckFailedException
+ // but retries were performed, attempt to reconcile the actual commit status.
+ if (!conditionCheckFailed || retryDetector.retried()) {
+ LOG.error(
+ "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
+ fullTableName,
+ persistFailure);
+ commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ }
+
+ // If we got ConditionalCheckFailedException, but find we
+ // succeeded on a retry that threw an exception, skip this exception.
+ if (commitStatus != CommitStatus.SUCCESS && conditionCheckFailed) {
+ throw new CommitFailedException(
+ persistFailure, "Cannot commit %s: concurrent update detected", tableName());
+ }
Review Comment:
Ah nvm that's still done below okay cool!
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java:
##########
@@ -340,6 +330,39 @@ void persistGlueTable(Table glueTable, Map<String, String> parameters, TableMeta
}
}
+ private void handleAWSExceptions(AwsServiceException persistFailure) {
+ int statusCode = persistFailure.statusCode();
Review Comment:
Very minor nit: Generally for better readability I try and having variables as close as possible to where they are read. I think we can move the status code to the else block or have some separate helper method `shouldThrow(AwsServiceexception)` which just returns based on the critera
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1153794647
##########
aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogCommitFailure.java:
##########
@@ -123,6 +128,74 @@ public void testConcurrentModificationExceptionDoesNotCheckCommitStatus() {
Assert.assertEquals("No new metadata files should exist", 2, metadataFileCount(ops.current()));
}
+ @Test
+ public void testCheckCommitStatusAfterRetries() {
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ TableMetadata metadataV1 = spyOps.current();
+ simulateRetriedCommit(spyOps, true);
+ updateTable(table, spyOps);
+
+ Assert.assertNotEquals("Current metadata should have changed", metadataV1, spyOps.current());
+ Assert.assertTrue("Current metadata should still exist", metadataFileExists(spyOps.current()));
+ Assert.assertEquals(
+ "No new metadata files should exist", 2, metadataFileCount(spyOps.current()));
+ }
+
+ @Test
+ public void testNoRetryAwarenessCorruptsTable() {
+ // This test exists to replicate the issue the prior test validates the fix for
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ simulateRetriedCommit(spyOps, false);
Review Comment:
You are correct that the goal of this PR is to make the situation this test replicates impossible. This test was created for 3 reasons:
* As a way of reliability reproducing the issue experienced in production
* As a form of monitor to confirm that the issue can still occur if the `RetryDetector` mechanism introduced is not made aware of the retry
* As a test of correct behavior from the `RetryDetector`. Though this could and arguably should be accomplished via separate set of unit tests for the `RetryDetector`, which I think I will add in a follow-up commit regardless of the outcome of this conversation.
If AWS Glue service or AWS SDK behavior changes such that this change is no longer necessary to prevent this failure mode from occurring (such as Glue making this API transparently idempotent), then this test would fail and we would know at that point we needed to re-evaluate the mechanism.
While I plan to address the 3rd reason via a unit test, I still believe the second reason in particular provides ongoing value as an integ test when dealing with software components we don't control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1157413800
##########
aws/src/main/java/org/apache/iceberg/aws/util/RetryDetector.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.util;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import software.amazon.awssdk.core.metrics.CoreMetric;
+import software.amazon.awssdk.metrics.MetricCollection;
+import software.amazon.awssdk.metrics.MetricPublisher;
+
+/**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. This class can be attached to an AWS API call and checked after to determine if retries
+ * occurred.
+ */
+public class RetryDetector implements MetricPublisher {
Review Comment:
So the main problem is this is used by both `GlueTableOperations` and `DynamoDbTableOperations` which are in different packages. Unfortunately, package-private visibility in Java does not make it visible to sub-packages. I originally attempted to hide this as much as possible given that limitation, as discussed [here](https://github.com/apache/iceberg/pull/7198#discussion_r1151191961), by both calling it `InternalRetryDetector` and hiding it inside of `AwsClientFactories`, but was asked not to do either of those.
With those limitations in mind, the only other way to avoid making this public would be to make it a `protected` method of a superclass of `GlueTableOperations` and `DynamoDbTableOperations`. At present, their common superclass is `BaseMetastoreTableOperations` which is not part of the aws module. As a result, taking this approach would require creating an intermediate superclass that inherits from `BaseMetastoreTableOperations` which `GlueTableOperations` and `DynamoDbTableOperations` inherit from instead. This is too much additional complexity to work around a visibility concern.
Unfortunately, the Iceberg project doesn't have a convention for declaring a public class as "not for public use" like Flink's `@Internal` annotation or Spark's `.internal.` package infix, so that wasn't an option.
I am not going to change this for now due to lack of options, but am open to further suggestions in this comment chain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ryanyuan commented on pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ryanyuan (via GitHub)" <gi...@apache.org>.
ryanyuan commented on PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#issuecomment-1504325049
Thanks @ChristinaTech.
@jackye1995 @amogh-jahagirdar @JonasJ-ap Could you please include this fix in the next release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1152511322
##########
aws/src/integration/java/org/apache/iceberg/aws/glue/GlueTestBase.java:
##########
@@ -83,13 +82,7 @@ public static void beforeClass() {
AwsProperties properties = new AwsProperties();
properties.setS3FileIoDeleteBatchSize(10);
glueCatalog.initialize(
- catalogName,
- testBucketPath,
- properties,
- glue,
- LockManagers.defaultLockManager(),
- fileIO,
- ImmutableMap.of());
+ catalogName, testBucketPath, properties, glue, null, fileIO, ImmutableMap.of());
Review Comment:
Yes we can remove it, it's already legacy feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JonasJ-ap commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "JonasJ-ap (via GitHub)" <gi...@apache.org>.
JonasJ-ap commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1154751816
##########
aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogCommitFailure.java:
##########
@@ -123,6 +128,74 @@ public void testConcurrentModificationExceptionDoesNotCheckCommitStatus() {
Assert.assertEquals("No new metadata files should exist", 2, metadataFileCount(ops.current()));
}
+ @Test
+ public void testCheckCommitStatusAfterRetries() {
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ TableMetadata metadataV1 = spyOps.current();
+ simulateRetriedCommit(spyOps, true);
+ updateTable(table, spyOps);
+
+ Assert.assertNotEquals("Current metadata should have changed", metadataV1, spyOps.current());
+ Assert.assertTrue("Current metadata should still exist", metadataFileExists(spyOps.current()));
+ Assert.assertEquals(
+ "No new metadata files should exist", 2, metadataFileCount(spyOps.current()));
+ }
+
+ @Test
+ public void testNoRetryAwarenessCorruptsTable() {
+ // This test exists to replicate the issue the prior test validates the fix for
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ simulateRetriedCommit(spyOps, false);
Review Comment:
Thank you for your clarification! +1 for unit tests on `RetryDetector`
> If AWS Glue service or AWS SDK behavior changes such that this change is no longer necessary to prevent this failure mode from occurring (such as Glue making this API transparently idempotent), then this test would fail and we would know at that point we needed to re-evaluate the mechanism.
I can now fully understand the purpose of the test. I have a concern regarding its failure conditions. In general, test failures usually indicate bugs or errors in code functionality. However, in this particular case, it seems that even if Glue were to update this API to be idempotent in the future, the existing mechanism would still work (otherwise `testCheckCommitStatusAfterRetries` would also fail). This means that the failure of this test alone does not necessarily indicate that there is something wrong with the mechanism that needs to be revised, but rather that it could be improved for better performance.
I think if we want to add this test, we may want to add more descriptive language to the test and explicitly reflect its purpose in the error message. This will help clarify the intent of the test and avoid any potential misunderstandings for other developers who may run this test in the future. For example, we can have something like the following in the error message for this test:
```java
// The update to the Glue API has made it no longer necessary to detect retry and check commit.
// As a result, the table is successfully loaded.
// We may want to consider update the related mechanism in Glue/DynamoDB Catalog
Assertions.assertThatThrownBy(() -> glueCatalog.loadTable(tableId))
.as("Retry Detection and Commit Check Not Required")
...
```
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1157365296
##########
aws/src/main/java/org/apache/iceberg/aws/glue/GlueTableOperations.java:
##########
@@ -340,6 +330,39 @@ void persistGlueTable(Table glueTable, Map<String, String> parameters, TableMeta
}
}
+ private void handleAWSExceptions(AwsServiceException persistFailure) {
+ int statusCode = persistFailure.statusCode();
Review Comment:
I will refactor to put it as a separate check inside the else block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1157377212
##########
aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbTableOperations.java:
##########
@@ -116,20 +118,30 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
.build());
checkMetadataLocation(table, base);
Map<String, String> properties = prepareProperties(table, newMetadataLocation);
- persistTable(tableKey, table, properties);
+ persistTable(tableKey, table, properties, retryDetector);
commitStatus = CommitStatus.SUCCESS;
- } catch (ConditionalCheckFailedException e) {
- throw new CommitFailedException(
- e, "Cannot commit %s: concurrent update detected", tableName());
} catch (CommitFailedException e) {
// any explicit commit failures are passed up and out to the retry handler
throw e;
} catch (RuntimeException persistFailure) {
- LOG.error(
- "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
- fullTableName,
- persistFailure);
- commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ boolean conditionCheckFailed = persistFailure instanceof ConditionalCheckFailedException;
+
+ // If we got an exception we weren't expecting, or we got a ConditionalCheckFailedException
+ // but retries were performed, attempt to reconcile the actual commit status.
+ if (!conditionCheckFailed || retryDetector.retried()) {
+ LOG.error(
+ "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
+ fullTableName,
+ persistFailure);
+ commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ }
+
+ // If we got ConditionalCheckFailedException, but find we
+ // succeeded on a retry that threw an exception, skip this exception.
Review Comment:
If the comment risks causing more confusion than helping, I'll just remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1162075083
##########
aws/src/main/java/org/apache/iceberg/aws/util/RetryDetector.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.util;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import software.amazon.awssdk.core.metrics.CoreMetric;
+import software.amazon.awssdk.metrics.MetricCollection;
+import software.amazon.awssdk.metrics.MetricPublisher;
+
+/**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. This class can be attached to an AWS API call and checked after to determine if retries
+ * occurred.
+ */
+public class RetryDetector implements MetricPublisher {
Review Comment:
Thanks for the explanation @ChristinaTech yeah I agree with @jackye1995 this should be fine as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1151206976
##########
aws/src/main/java/org/apache/iceberg/aws/dynamodb/DynamoDbTableOperations.java:
##########
@@ -116,20 +119,30 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
.build());
checkMetadataLocation(table, base);
Map<String, String> properties = prepareProperties(table, newMetadataLocation);
- persistTable(tableKey, table, properties);
+ persistTable(tableKey, table, properties, retryDetector);
commitStatus = CommitStatus.SUCCESS;
- } catch (ConditionalCheckFailedException e) {
- throw new CommitFailedException(
- e, "Cannot commit %s: concurrent update detected", tableName());
} catch (CommitFailedException e) {
// any explicit commit failures are passed up and out to the retry handler
throw e;
} catch (RuntimeException persistFailure) {
- LOG.error(
- "Confirming if commit to {} indeed failed to persist, attempting to reconnect and check.",
- fullTableName,
- persistFailure);
- commitStatus = checkCommitStatus(newMetadataLocation, metadata);
+ boolean isConditionCheckFailedEx = persistFailure instanceof ConditionalCheckFailedException;
Review Comment:
nit: similar style comment, can avoid `is` and name it shorter, like `conditionCheckFailed`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1151191961
##########
aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java:
##########
@@ -88,6 +91,32 @@ private static AwsClientFactory loadClientFactory(String impl, Map<String, Strin
return factory;
}
+ /**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. Not for public use.
+ */
+ public static class InternalRetryDetector implements MetricPublisher {
Review Comment:
nit: for non-public classes, we should mark it as package protected or private
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1155044079
##########
aws/src/integration/java/org/apache/iceberg/aws/glue/TestGlueCatalogCommitFailure.java:
##########
@@ -123,6 +128,74 @@ public void testConcurrentModificationExceptionDoesNotCheckCommitStatus() {
Assert.assertEquals("No new metadata files should exist", 2, metadataFileCount(ops.current()));
}
+ @Test
+ public void testCheckCommitStatusAfterRetries() {
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ TableMetadata metadataV1 = spyOps.current();
+ simulateRetriedCommit(spyOps, true);
+ updateTable(table, spyOps);
+
+ Assert.assertNotEquals("Current metadata should have changed", metadataV1, spyOps.current());
+ Assert.assertTrue("Current metadata should still exist", metadataFileExists(spyOps.current()));
+ Assert.assertEquals(
+ "No new metadata files should exist", 2, metadataFileCount(spyOps.current()));
+ }
+
+ @Test
+ public void testNoRetryAwarenessCorruptsTable() {
+ // This test exists to replicate the issue the prior test validates the fix for
+ String namespace = createNamespace();
+ String tableName = createTable(namespace);
+ TableIdentifier tableId = TableIdentifier.of(namespace, tableName);
+
+ GlueTableOperations spyOps =
+ Mockito.spy((GlueTableOperations) glueCatalog.newTableOps(tableId));
+ GlueCatalog spyCatalog = Mockito.spy(glueCatalog);
+ Mockito.doReturn(spyOps).when(spyCatalog).newTableOps(Mockito.eq(tableId));
+ Table table = spyCatalog.loadTable(tableId);
+
+ simulateRetriedCommit(spyOps, false);
Review Comment:
I agree, I will add something similar to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#issuecomment-1483944062
Confirmed this doesn't break the existing integration tests. Going to add some additional ones to better replicate the scenario this is designed to fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1151196353
##########
aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java:
##########
@@ -88,6 +91,32 @@ private static AwsClientFactory loadClientFactory(String impl, Map<String, Strin
return factory;
}
+ /**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. Not for public use.
+ */
+ public static class InternalRetryDetector implements MetricPublisher {
+ private boolean retried = false;
+
+ @Override
+ public void publish(MetricCollection metricCollection) {
+ if (!retried) {
+ if (metricCollection.metricValues(CoreMetric.RETRY_COUNT).stream().anyMatch(i -> i > 0)) {
+ retried = true;
+ } else {
+ metricCollection.children().forEach(this::publish);
+ }
+ }
+ }
+
+ @Override
+ public void close() {}
+
+ public boolean wasRetried() {
Review Comment:
nit: prefer simple adjective method names for boolean, like just `retried()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1151194306
##########
aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java:
##########
@@ -88,6 +91,32 @@ private static AwsClientFactory loadClientFactory(String impl, Map<String, Strin
return factory;
}
+ /**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. Not for public use.
+ */
+ public static class InternalRetryDetector implements MetricPublisher {
+ private boolean retried = false;
Review Comment:
I remember that retries could happen in parallel. Should we make this an AtomicBoolean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ChristinaTech commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "ChristinaTech (via GitHub)" <gi...@apache.org>.
ChristinaTech commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1151225719
##########
aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java:
##########
@@ -88,6 +91,32 @@ private static AwsClientFactory loadClientFactory(String impl, Map<String, Strin
return factory;
}
+ /**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. Not for public use.
+ */
+ public static class InternalRetryDetector implements MetricPublisher {
+ private boolean retried = false;
Review Comment:
I don't believe parallel retries are done by the non-async client, but on the off chance you are right, or become right in the future, it doesn't hurt to make this an `AtomicBoolean`, so will change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7198: AWS: Check commit status after failed commit if AWS client performed retries
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7198:
URL: https://github.com/apache/iceberg/pull/7198#discussion_r1161873678
##########
aws/src/main/java/org/apache/iceberg/aws/util/RetryDetector.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.aws.util;
+
+import java.util.concurrent.atomic.AtomicBoolean;
+import software.amazon.awssdk.core.metrics.CoreMetric;
+import software.amazon.awssdk.metrics.MetricCollection;
+import software.amazon.awssdk.metrics.MetricPublisher;
+
+/**
+ * Metrics are the only reliable way provided by the AWS SDK to determine if an API call was
+ * retried. This class can be attached to an AWS API call and checked after to determine if retries
+ * occurred.
+ */
+public class RetryDetector implements MetricPublisher {
Review Comment:
Given the fact that it has to be shared across different classes, and potentially attached to other things in the future, I am fine with it being public. We don't usually use Internal annotation in Iceberg, we can keep it as is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org