You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amogh-jahagirdar (via GitHub)" <gi...@apache.org> on 2024/04/15 00:19:41 UTC
[PR] Retry connections in JDBC catalog [iceberg]
amogh-jahagirdar opened a new pull request, #10140:
URL: https://github.com/apache/iceberg/pull/10140
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "cccs-eric (via GitHub)" <gi...@apache.org>.
cccs-eric commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2056558173
@amogh-jahagirdar I'd be happy with this fix since it will solve the problem my organization is facing. Thanks for looking into this and bringing the football past the goal line. 👍🏻
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1588457797
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java:
##########
@@ -40,6 +40,8 @@ final class JdbcUtil {
// property to control if view support is added to the existing database
static final String SCHEMA_VERSION_PROPERTY = JdbcCatalog.PROPERTY_PREFIX + "schema-version";
+ static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
> that anything specific to JDBC catalog should be prefixed by jdbc, just to be sure we don't have conflict with properties with same name not related to JDBC Catalog. As JDBC Connection is used in JDBC Catalog, I would prefix with jdbc for consistency. As it's user facing properties, it's important to be consistent here.
I think if we do this it's not really consistent with what we do with other catalogs. For example, REST, Glue, Nessie catalogs don't neccessarily prefix any of their custom configurations with "glue", "rest", or "nessie" respectively; configuring
the catalog configuration is just a matter of setting the property via <catalog_name>.<property>. I also don't think we need to worry about conflicts in property names across different catalogs; in the end we use reflection to load the catalog and it's up to the catalog implementation to respect whichever properties we want, so we just need to make sure the properties are well defined.
The docs seemed pretty clear to me about the original intention of the "jdbc" prefix for the JDBC catalog https://iceberg.apache.org/docs/nightly/jdbc/?h=jdbc#configurations is that it's specifically meant for configuring JDBC connections (username/port/db name that are in the URL itself).
`retryable_status_codes` is *not* configuring the JDBC connection URL.
It is a catalog level property which controls when the catalog should retry establishing an entirely new connection. That's why I think it should be namespaced as it's own property (same as any other property that we do for other catalogs).
I feel like that's actually more consistent behavior for a user; they can know that everything under jdbc is part of the actual connection URL, and everything outside of the JDBC prefix is a general catalog level property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581120323
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map<String, String> props) {
}
public JdbcClientPool(int poolSize, String dbUrl, Map<String, String> props) {
- super(poolSize, SQLNonTransientConnectionException.class, true);
+ super(poolSize, SQLTransientException.class, true);
Review Comment:
using `SQLTransientException` here looks correct to me, since the javadoc of `SQLNonTransientConnectionException` says
```
The subclass of SQLException thrown for the SQLState class value '08', or under
vendor-specified conditions. This indicates that the connection operation that
failed will not succeed if the operation is
retried without the cause of the failure being corrected.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565006119
##########
core/src/main/java/org/apache/iceberg/ClientPoolImpl.java:
##########
@@ -56,26 +67,36 @@ public <R> R run(Action<R, C, E> action, boolean retry) throws E, InterruptedExc
C client = get();
try {
return action.run(client);
-
} catch (Exception exc) {
- if (retry && isConnectionException(exc)) {
- try {
- client = reconnect(client);
- } catch (Exception ignored) {
- // if reconnection throws any exception, rethrow the original failure
- throw reconnectExc.cast(exc);
- }
-
- return action.run(client);
+ if (!isConnectionException(exc)) {
+ throw exc;
}
- throw exc;
+ return retryAction(action, exc, client);
} finally {
release(client);
}
}
+ private <R> R retryAction(Action<R, C, E> action, Exception originalFailure, C client) throws E {
+ int retryAttempts = 0;
+ while (retryAttempts < maxRetries) {
+ try {
+ C reconnectedClient = reconnect(client);
+ return action.run(reconnectedClient);
+ } catch (Exception exc) {
+ if (isConnectionException(exc)) {
+ retryAttempts++;
+ } else {
+ throw reconnectExc.cast(originalFailure);
+ }
+ }
+ }
+
+ throw reconnectExc.cast(originalFailure);
Review Comment:
I wanted to use `Tasks` for all this :(, but it's seemingly infeasible without passing through the actual class for `E` so that we can handle that appropriately during task failure (the generic E and the handling at Tasks are just impossible to reconcile without fundamental API changes). That is kind of a hack and also adds an unneccessary API change. I think the best option is just to write the retry logic.
##########
core/src/main/java/org/apache/iceberg/ClientPoolImpl.java:
##########
@@ -56,26 +67,36 @@ public <R> R run(Action<R, C, E> action, boolean retry) throws E, InterruptedExc
C client = get();
try {
return action.run(client);
-
} catch (Exception exc) {
- if (retry && isConnectionException(exc)) {
- try {
- client = reconnect(client);
- } catch (Exception ignored) {
- // if reconnection throws any exception, rethrow the original failure
- throw reconnectExc.cast(exc);
- }
-
- return action.run(client);
+ if (!isConnectionException(exc)) {
+ throw exc;
}
- throw exc;
+ return retryAction(action, exc, client);
} finally {
release(client);
}
}
+ private <R> R retryAction(Action<R, C, E> action, Exception originalFailure, C client) throws E {
+ int retryAttempts = 0;
+ while (retryAttempts < maxRetries) {
+ try {
+ C reconnectedClient = reconnect(client);
+ return action.run(reconnectedClient);
+ } catch (Exception exc) {
+ if (isConnectionException(exc)) {
+ retryAttempts++;
+ } else {
+ throw reconnectExc.cast(originalFailure);
+ }
+ }
+ }
+
+ throw reconnectExc.cast(originalFailure);
Review Comment:
Maybe add some backoff/jitter?
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map<String, String> props) {
}
public JdbcClientPool(int poolSize, String dbUrl, Map<String, String> props) {
- super(poolSize, SQLNonTransientConnectionException.class, true);
+ super(poolSize, SQLTransientException.class, true);
properties = props;
+ retryableStatusCodes = Sets.newHashSet();
+ retryableStatusCodes.addAll(COMMON_RETRYABLE_CONNECTION_SQL_STATES);
+ String configuredRetryableStatuses = props.get(JdbcCatalog.RETRYABLE_STATUS_CODES);
+ if (configuredRetryableStatuses != null) {
Review Comment:
I'm presenting this configuration option since at the moment I believe it's the most practical way for people to work around production issues they hit today.
Long term, if we separate JDBC catalog into a separate library (I forgot where we landed on where JDBC fits in in terms of the catalog discussion that's happened in the past) that library could define status codes for each database so a user doesn't have to specify anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565280613
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java:
##########
@@ -72,6 +72,7 @@ public class JdbcCatalog extends BaseMetastoreViewCatalog
implements Configurable<Object>, SupportsNamespaces {
public static final String PROPERTY_PREFIX = "jdbc.";
+ public static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
would be good to move this, as properties are defined in `JdbcUtil`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #10140:
URL: https://github.com/apache/iceberg/pull/10140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581120323
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map<String, String> props) {
}
public JdbcClientPool(int poolSize, String dbUrl, Map<String, String> props) {
- super(poolSize, SQLNonTransientConnectionException.class, true);
+ super(poolSize, SQLTransientException.class, true);
Review Comment:
using `SQLTransientException` here looks correct to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851587
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map<String, String> props) {
}
public JdbcClientPool(int poolSize, String dbUrl, Map<String, String> props) {
- super(poolSize, SQLNonTransientConnectionException.class, true);
+ super(poolSize, SQLTransientException.class, true);
Review Comment:
Yup! I think it was just a bug since the beginning, figured we may as well change the default here since we're addressing retries in generla.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565313138
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -21,17 +21,27 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTransientException;
+import java.util.Arrays;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.ClientPoolImpl;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
public class JdbcClientPool extends ClientPoolImpl<Connection, SQLException> {
private final String dbUrl;
private final Map<String, String> properties;
+ private Set<String> retryableStatusCodes;
+
+ private static final Set<String> COMMON_RETRYABLE_CONNECTION_SQL_STATES =
Review Comment:
Maybe make it by default but also configurable (not necessarily in this PR). For instance, in ActiveMQ JDBC Store, I did some special handling specific to vendor (some SQL states are specific to vendor). I can include that in my PR (in preparation) working with multiple vendors.
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java:
##########
@@ -72,6 +72,7 @@ public class JdbcCatalog extends BaseMetastoreViewCatalog
implements Configurable<Object>, SupportsNamespaces {
public static final String PROPERTY_PREFIX = "jdbc.";
+ public static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
To be consistent, I would move this property in `JdbcUtil` as it's where all properties are located.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851517
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map<String, String> props) {
}
public JdbcClientPool(int poolSize, String dbUrl, Map<String, String> props) {
- super(poolSize, SQLNonTransientConnectionException.class, true);
+ super(poolSize, SQLTransientException.class, true);
properties = props;
+ retryableStatusCodes = Sets.newHashSet();
+ retryableStatusCodes.addAll(COMMON_RETRYABLE_CONNECTION_SQL_STATES);
+ String configuredRetryableStatuses = props.get(JdbcUtil.RETRYABLE_STATUS_CODES);
+ if (configuredRetryableStatuses != null) {
+ retryableStatusCodes.addAll(
+ Arrays.stream(configuredRetryableStatuses.split(",")).collect(Collectors.toSet()));
Review Comment:
Fixed to handle this case!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851445
##########
core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java:
##########
@@ -167,6 +168,67 @@ public void testInitialize() {
jdbcCatalog.initialize("test_jdbc_catalog", properties);
}
+ @Test
+ public void testRetryingErrorCodesProperty() {
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(CatalogProperties.WAREHOUSE_LOCATION, this.tableDir.toAbsolutePath().toString());
+ properties.put(CatalogProperties.URI, "jdbc:sqlite:file::memory:?icebergDB");
+ properties.put(JdbcUtil.RETRYABLE_STATUS_CODES, "57000,57P03,57P04");
Review Comment:
Updated to handle allowing users passing in a space in the comma delimited list (we just strip the spaces)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "guyco33 (via GitHub)" <gi...@apache.org>.
guyco33 commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2143850454
@amogh-jahagirdar When is it going to be released ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "ochanism (via GitHub)" <gi...@apache.org>.
ochanism commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2154961273
Oh, I also need this feature for my REST/JDBC catalog. I've been suffering from the JDBC connection failure issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565013391
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map<String, String> props) {
}
public JdbcClientPool(int poolSize, String dbUrl, Map<String, String> props) {
- super(poolSize, SQLNonTransientConnectionException.class, true);
+ super(poolSize, SQLTransientException.class, true);
properties = props;
+ retryableStatusCodes = Sets.newHashSet();
+ retryableStatusCodes.addAll(COMMON_RETRYABLE_CONNECTION_SQL_STATES);
+ String configuredRetryableStatuses = props.get(JdbcCatalog.RETRYABLE_STATUS_CODES);
+ if (configuredRetryableStatuses != null) {
Review Comment:
I'm presenting this configuration option since at the moment I believe it's the most practical way for people to work around production issues they hit today.
Long term, if we separate JDBC catalog into a separate library (I forgot where we landed on where JDBC fits in in terms of the catalog discussion that's happened in the past) that library could define status codes for each common, industry-standard database so a user doesn't have to specify anything.
See https://www.youtube.com/watch?v=uAQVGd5zV4I&t=1323s for the discussion on catalogs in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2055350705
cc: @jbonofre
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1584047406
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ static class RetryableException extends RuntimeException {}
+
+ static class NonRetryableException extends RuntimeException {}
+
+ static class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ static class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
Review Comment:
Good suggestion, I agreed the tests are more readable that way!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581127958
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ class RetryableException extends RuntimeException {}
+
+ class NonRetryableException extends RuntimeException {}
+
+ class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
+ public void testRetrySucceedsWithinMaxAttempts() {
+ int maxRetries = 5;
+ int succeedAfterAttempts = 3;
+ try (MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries)) {
+ int actions = mockClientPool.run(client -> client.succeedAfter(succeedAfterAttempts));
+ Assertions.assertThat(actions)
+ .as("There should be exactly one successful action invocation")
+ .isEqualTo(1);
+ Assertions.assertThat(mockClientPool.reconnectionAttempts())
+ .isEqualTo(succeedAfterAttempts - 1);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ public void testRetriesExhaustedAndSurfacesFailure() {
+ int maxRetries = 3;
+ int succeedAfterAttempts = 5;
+ MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries);
+ assertThrows(
Review Comment:
we typically use `assertThatThrownBy(..)` from AssertJ rather than the stuff from JUnit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "guyco33 (via GitHub)" <gi...@apache.org>.
guyco33 commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2143888421
Thanks @jbonofre
It would be great to have it earlier as cherry picking patch of [1.5.x](https://github.com/apache/iceberg/tree/1.5.x) on top of [pr-10124](https://github.com/apache/iceberg/pull/10124) that is needed also to avoid conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2104984953
I'll go ahead and merge this since I think we have consensus on https://github.com/apache/iceberg/pull/10140#discussion_r1581123625. Thanks for reviewing @nastra @jbonofre !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2054248088
Need to add tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2076302600
I didn't forget about this (again :) ) I've added a test to verify the error code configuration get parsed and considered correctly and am adding tests for ClientPoolImpl which we can have a mocked implementation (mocking the actual connection action and failure cases) to verify the new retry path works as expected. This combination of tests should give us confidence on the real behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851484
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ class RetryableException extends RuntimeException {}
+
+ class NonRetryableException extends RuntimeException {}
+
+ class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
+ public void testRetrySucceedsWithinMaxAttempts() {
+ int maxRetries = 5;
+ int succeedAfterAttempts = 3;
+ try (MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries)) {
+ int actions = mockClientPool.run(client -> client.succeedAfter(succeedAfterAttempts));
+ Assertions.assertThat(actions)
+ .as("There should be exactly one successful action invocation")
+ .isEqualTo(1);
+ Assertions.assertThat(mockClientPool.reconnectionAttempts())
+ .isEqualTo(succeedAfterAttempts - 1);
+ } catch (Exception e) {
Review Comment:
Yeah I forgot why I had this here, it's not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581124122
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map<String, String> props) {
}
public JdbcClientPool(int poolSize, String dbUrl, Map<String, String> props) {
- super(poolSize, SQLNonTransientConnectionException.class, true);
+ super(poolSize, SQLTransientException.class, true);
properties = props;
+ retryableStatusCodes = Sets.newHashSet();
+ retryableStatusCodes.addAll(COMMON_RETRYABLE_CONNECTION_SQL_STATES);
+ String configuredRetryableStatuses = props.get(JdbcUtil.RETRYABLE_STATUS_CODES);
+ if (configuredRetryableStatuses != null) {
+ retryableStatusCodes.addAll(
+ Arrays.stream(configuredRetryableStatuses.split(",")).collect(Collectors.toSet()));
Review Comment:
we might need to make sure that users don't accidentally pass this with spaces: `123, 456, 789`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1589245337
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java:
##########
@@ -40,6 +40,8 @@ final class JdbcUtil {
// property to control if view support is added to the existing database
static final String SCHEMA_VERSION_PROPERTY = JdbcCatalog.PROPERTY_PREFIX + "schema-version";
+ static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
Fair enough. It makes sense indeed. But it also means that `strict-mode` and `schema-version` should not have been prefixed by `jdbc` then.
We can change that, but as it's user facing, it could be problematic.
I can work on a PR to deprecated `jdbc` prefix (not still supporting it) for `strict-mode` and `schema-version`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "palkx (via GitHub)" <gi...@apache.org>.
palkx commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2203230368
Thank you for the update! Will wait for the v1.6.0 then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581695111
##########
core/src/main/java/org/apache/iceberg/ClientPoolImpl.java:
##########
@@ -56,26 +69,38 @@ public <R> R run(Action<R, C, E> action, boolean retry) throws E, InterruptedExc
C client = get();
try {
return action.run(client);
-
} catch (Exception exc) {
- if (retry && isConnectionException(exc)) {
Review Comment:
Good catch, I was getting lucky in the `testNoRetryingWhenDisabled` I added since the retries I passed in was 0. If the flag is false we should never retry .
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java:
##########
@@ -40,6 +40,8 @@ final class JdbcUtil {
// property to control if view support is added to the existing database
static final String SCHEMA_VERSION_PROPERTY = JdbcCatalog.PROPERTY_PREFIX + "schema-version";
+ static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
I don't think so , looking at https://iceberg.apache.org/docs/nightly/jdbc/?h=jdbc#iceberg-jdbc-integration
The properties which are prefixed with "jdbc" look to be specifically for configuring the JDBC connection url, like user,password, ssl etc. These are standard options when establishing a JDBC connection.
What we're trying to do here is independent of the JDBC connection URL and more like a typical catalog property.
I do see that we have strict-mode and schema-version catalog properties which kind of break this pattern since those aren't relevant for the JDBC connection URL, but I'm not sure we want to keep following that since it does seem like the original intention of the prefix was pretty clear on having it be for the connection URL. cc @jbonofre if you have a take on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581117582
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -21,17 +21,27 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTransientException;
+import java.util.Arrays;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.ClientPoolImpl;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
public class JdbcClientPool extends ClientPoolImpl<Connection, SQLException> {
private final String dbUrl;
private final Map<String, String> properties;
+ private Set<String> retryableStatusCodes;
+
+ private static final Set<String> COMMON_RETRYABLE_CONNECTION_SQL_STATES =
Review Comment:
I think it would be helpful to add some javadoc and document what those errors actually mean.
nit: the definition of that var should probably be moved to the top (before the non-static vars)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1584048101
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ static class RetryableException extends RuntimeException {}
+
+ static class NonRetryableException extends RuntimeException {}
+
+ static class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ static class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
+ public void testRetrySucceedsWithinMaxAttempts() throws Exception {
+ int maxRetries = 5;
+ int succeedAfterAttempts = 3;
+ try (MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries)) {
+ int actions = mockClientPool.run(client -> client.succeedAfter(succeedAfterAttempts));
+ assertThat(actions)
+ .as("There should be exactly one successful action invocation")
+ .isEqualTo(1);
+ assertThat(mockClientPool.reconnectionAttempts()).isEqualTo(succeedAfterAttempts - 1);
+ }
+ }
+
+ @Test
+ public void testRetriesExhaustedAndSurfacesFailure() {
+ int maxRetries = 3;
+ int succeedAfterAttempts = 5;
+ MockClientPoolImpl mockClientPool =
Review Comment:
Yes we should. There's no real resources being allocated for the mock stuff but it's still best practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1588457797
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java:
##########
@@ -40,6 +40,8 @@ final class JdbcUtil {
// property to control if view support is added to the existing database
static final String SCHEMA_VERSION_PROPERTY = JdbcCatalog.PROPERTY_PREFIX + "schema-version";
+ static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
> that anything specific to JDBC catalog should be prefixed by jdbc, just to be sure we don't have conflict with properties with same name not related to JDBC Catalog. As JDBC Connection is used in JDBC Catalog, I would prefix with jdbc for consistency. As it's user facing properties, it's important to be consistent here.
I think if we do this it's not really consistent with what we do with other catalogs. For example, REST, Glue, Nessie catalogs don't neccessarily prefix any of their custom configurations with "glue", "rest", or "nessie" respectively; configuring
the catalog configuration is just a matter of setting the property via `<catalog_name>.<property>`. I also don't think we need to worry about conflicts in property names across different catalogs; in the end we use reflection to load the catalog and it's up to the catalog implementation to respect whichever properties we want, so we just need to make sure the properties are well defined.
The docs seemed pretty clear to me about the original intention of the "jdbc" prefix for the JDBC catalog https://iceberg.apache.org/docs/nightly/jdbc/?h=jdbc#configurations is that it's specifically meant for configuring JDBC connections (username/port/db name that are in the URL itself).
`retryable_status_codes` is *not* configuring the JDBC connection URL.
It is a catalog level property which controls when the catalog should retry establishing an entirely new connection. That's why I think it should be namespaced as it's own property (same as any other property that we do for other catalogs).
I feel like that's actually more consistent behavior for a user; they can know that everything under jdbc is part of the actual connection URL, and everything outside of the JDBC prefix is a general catalog level property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1582748026
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ static class RetryableException extends RuntimeException {}
+
+ static class NonRetryableException extends RuntimeException {}
+
+ static class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ static class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
Review Comment:
I would probably move the tests to the top and the inner classes to the bottom
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565848626
##########
core/src/main/java/org/apache/iceberg/ClientPoolImpl.java:
##########
@@ -56,26 +67,36 @@ public <R> R run(Action<R, C, E> action, boolean retry) throws E, InterruptedExc
C client = get();
try {
return action.run(client);
-
} catch (Exception exc) {
- if (retry && isConnectionException(exc)) {
- try {
- client = reconnect(client);
- } catch (Exception ignored) {
- // if reconnection throws any exception, rethrow the original failure
- throw reconnectExc.cast(exc);
- }
-
- return action.run(client);
+ if (!isConnectionException(exc)) {
+ throw exc;
}
- throw exc;
+ return retryAction(action, exc, client);
} finally {
release(client);
}
}
+ private <R> R retryAction(Action<R, C, E> action, Exception originalFailure, C client) throws E {
+ int retryAttempts = 0;
+ while (retryAttempts < maxRetries) {
+ try {
+ C reconnectedClient = reconnect(client);
+ return action.run(reconnectedClient);
+ } catch (Exception exc) {
+ if (isConnectionException(exc)) {
+ retryAttempts++;
+ } else {
+ throw reconnectExc.cast(originalFailure);
+ }
+ }
+ }
+
+ throw reconnectExc.cast(originalFailure);
Review Comment:
So my thoughts @jbonofre:
1.) C3P0 is a great pooling library! Less familiar with DBCP2 but I'm sure it's in a similar vein. We should leverage such libraries where we can to avoid reimplementing this sort of complex, redundant logic.
2.) That being said, given that long term we would be separating out the catalog into a separate project from what we discussed in the past community sync, the question comes down to (I think) :
do we want to do the work of adding the C3P0 dependency, changing the code to work with that at this point in time, or do we want to do it after a migration to a separate project?
I fall in the latter camp since then we'd need to maintain the implementation until the migration were to happen; and we've already reached a point where it's (imo) a lot for the Iceberg project itself to maintain. The improvements for multiple backends just working is great, since that's what people expect from JDBC but again I'm not sure if we want to that upfront here or just a separate catalog project which we would do just prior to a 2.0 version of Iceberg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1582745688
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ static class RetryableException extends RuntimeException {}
+
+ static class NonRetryableException extends RuntimeException {}
+
+ static class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ static class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
+ public void testRetrySucceedsWithinMaxAttempts() throws Exception {
+ int maxRetries = 5;
+ int succeedAfterAttempts = 3;
+ try (MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries)) {
+ int actions = mockClientPool.run(client -> client.succeedAfter(succeedAfterAttempts));
+ assertThat(actions)
+ .as("There should be exactly one successful action invocation")
+ .isEqualTo(1);
+ assertThat(mockClientPool.reconnectionAttempts()).isEqualTo(succeedAfterAttempts - 1);
+ }
+ }
+
+ @Test
+ public void testRetriesExhaustedAndSurfacesFailure() {
+ int maxRetries = 3;
+ int succeedAfterAttempts = 5;
+ MockClientPoolImpl mockClientPool =
Review Comment:
should this (and other places) be in a try-with-resources block like in the other test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1584397889
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -21,17 +21,39 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTransientException;
+import java.util.Arrays;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.ClientPoolImpl;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
public class JdbcClientPool extends ClientPoolImpl<Connection, SQLException> {
+ /**
+ * The following are common retryable SQLSTATEs error codes which are generic across vendors.
Review Comment:
```suggestion
* The following are common retryable SQLSTATE error codes which are generic across vendors.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581114289
##########
core/src/main/java/org/apache/iceberg/ClientPoolImpl.java:
##########
@@ -56,26 +69,38 @@ public <R> R run(Action<R, C, E> action, boolean retry) throws E, InterruptedExc
C client = get();
try {
return action.run(client);
-
} catch (Exception exc) {
- if (retry && isConnectionException(exc)) {
Review Comment:
it looks like the `retry` param is now being ignored
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1588457797
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java:
##########
@@ -40,6 +40,8 @@ final class JdbcUtil {
// property to control if view support is added to the existing database
static final String SCHEMA_VERSION_PROPERTY = JdbcCatalog.PROPERTY_PREFIX + "schema-version";
+ static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
> that anything specific to JDBC catalog should be prefixed by jdbc, just to be sure we don't have conflict with properties with same name not related to JDBC Catalog. As JDBC Connection is used in JDBC Catalog, I would prefix with jdbc for consistency. As it's user facing properties, it's important to be consistent here.
I think if we do this it's not really consistent with what we do with other catalogs. For example, REST, Glue, Nessie catalogs don't neccessarily prefix any of their custom configurations with "glue", "rest", or "nessie" respectively; configuring
the catalog configuration is just a matter of setting the property via <catalog_name>.<property>. I also don't think we need to worry about conflicts in property names across different catalogs; in the end we use reflection to load the catalog and it's up to the catalog implementation to respect whichever properties we want, so we just need to make sure the properties are well defined.
The docs seemed pretty clear to me about the original intention of the "jdbc" prefix for the JDBC catalog https://iceberg.apache.org/docs/nightly/jdbc/?h=jdbc#configurations is that it's specifically meant for configuring JDBC connections (username/port/db name that are in the URL itself).
`retryable_status_codes` is *not* configuring the JDBC connection URL.
It is a catalog level property which controls when the catalog should retry establishing an entirely new connection so that's why I think it should be namespaced as it's own property (same as any other property that we do for other catalogs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#issuecomment-2143854070
@guyco33 It will be included in Iceberg 1.6.0, plan in a few weeks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565208210
##########
core/src/main/java/org/apache/iceberg/ClientPoolImpl.java:
##########
@@ -56,26 +67,36 @@ public <R> R run(Action<R, C, E> action, boolean retry) throws E, InterruptedExc
C client = get();
try {
return action.run(client);
-
} catch (Exception exc) {
- if (retry && isConnectionException(exc)) {
- try {
- client = reconnect(client);
- } catch (Exception ignored) {
- // if reconnection throws any exception, rethrow the original failure
- throw reconnectExc.cast(exc);
- }
-
- return action.run(client);
+ if (!isConnectionException(exc)) {
+ throw exc;
}
- throw exc;
+ return retryAction(action, exc, client);
} finally {
release(client);
}
}
+ private <R> R retryAction(Action<R, C, E> action, Exception originalFailure, C client) throws E {
+ int retryAttempts = 0;
+ while (retryAttempts < maxRetries) {
+ try {
+ C reconnectedClient = reconnect(client);
+ return action.run(reconnectedClient);
+ } catch (Exception exc) {
+ if (isConnectionException(exc)) {
+ retryAttempts++;
+ } else {
+ throw reconnectExc.cast(originalFailure);
+ }
+ }
+ }
+
+ throw reconnectExc.cast(originalFailure);
Review Comment:
I wonder why we can't use a DBCP2 or C3P0 to deal with connection. They provide good pooling capabilities and retry/validate mechanism.
I just wonder if we explore this option (it's what I've started in the JDBC catalog refactoring using multiple backends support).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581125222
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ class RetryableException extends RuntimeException {}
+
+ class NonRetryableException extends RuntimeException {}
+
+ class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
+ public void testRetrySucceedsWithinMaxAttempts() {
+ int maxRetries = 5;
+ int succeedAfterAttempts = 3;
+ try (MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries)) {
+ int actions = mockClientPool.run(client -> client.succeedAfter(succeedAfterAttempts));
+ Assertions.assertThat(actions)
Review Comment:
just FYI, for new tests it's ok to statically import `assertThat`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581129079
##########
core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java:
##########
@@ -167,6 +168,67 @@ public void testInitialize() {
jdbcCatalog.initialize("test_jdbc_catalog", properties);
}
+ @Test
+ public void testRetryingErrorCodesProperty() {
+ Map<String, String> properties = Maps.newHashMap();
+ properties.put(CatalogProperties.WAREHOUSE_LOCATION, this.tableDir.toAbsolutePath().toString());
+ properties.put(CatalogProperties.URI, "jdbc:sqlite:file::memory:?icebergDB");
+ properties.put(JdbcUtil.RETRYABLE_STATUS_CODES, "57000,57P03,57P04");
Review Comment:
we might want to double-check here in case users pass this with a space after the comma
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1587385129
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -43,8 +65,18 @@ public JdbcClientPool(String dbUrl, Map<String, String> props) {
}
public JdbcClientPool(int poolSize, String dbUrl, Map<String, String> props) {
- super(poolSize, SQLNonTransientConnectionException.class, true);
+ super(poolSize, SQLTransientException.class, true);
properties = props;
+ retryableStatusCodes = Sets.newHashSet();
+ retryableStatusCodes.addAll(COMMON_RETRYABLE_CONNECTION_SQL_STATES);
+ String configuredRetryableStatuses = props.get(JdbcUtil.RETRYABLE_STATUS_CODES);
Review Comment:
Thanks to make it configuration (as discussed in the PR): it will allow users to define database specific states.
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -21,17 +21,39 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTransientException;
+import java.util.Arrays;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.ClientPoolImpl;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
public class JdbcClientPool extends ClientPoolImpl<Connection, SQLException> {
+ /**
+ * The following are common retryable SQLSTATE error codes which are generic across vendors.
+ *
+ * <ul>
+ * <li>08000: Generic Connection Exception
+ * <li>08003: Connection does not exist
+ * <li>08006: Connection failure
+ * <li>08007: Transaction resolution unknown
+ * </ul>
+ *
+ * See https://en.wikipedia.org/wiki/SQLSTATE for more details.
+ */
+ static final Set<String> COMMON_RETRYABLE_CONNECTION_SQL_STATES =
Review Comment:
Maybe worth to add `40001` and `40P01` state to the list:
- `40001` is the code used by most databases to identify deadlocks
- `40P01` is a PostgreSQL specific code for deadlocks as well (see https://www.postgresql.org/docs/13/errcodes-appendix.html)
Deadlocks are a candidate to retry imho.
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java:
##########
@@ -40,6 +40,8 @@ final class JdbcUtil {
// property to control if view support is added to the existing database
static final String SCHEMA_VERSION_PROPERTY = JdbcCatalog.PROPERTY_PREFIX + "schema-version";
+ static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
My take on this (maybe I'm wrong) is that anything specific to JDBC catalog should be prefixed by `jdbc`, just to be sure we don't have conflict with properties with same name not related to JDBC Catalog.
As JDBC Connection is used in JDBC Catalog, I would prefix with `jdbc` for consistency. As it's user facing properties, it's important to be consistent here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1588457797
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java:
##########
@@ -40,6 +40,8 @@ final class JdbcUtil {
// property to control if view support is added to the existing database
static final String SCHEMA_VERSION_PROPERTY = JdbcCatalog.PROPERTY_PREFIX + "schema-version";
+ static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
> that anything specific to JDBC catalog should be prefixed by jdbc, just to be sure we don't have conflict with properties with same name not related to JDBC Catalog. As JDBC Connection is used in JDBC Catalog, I would prefix with jdbc for consistency. As it's user facing properties, it's important to be consistent here.
I think if we do this it's not really consistent with what we do with other catalogs. For example, REST, Glue, Nessie catalogs don't prefix any of their custom configurations with "glue", "rest", or "nessie" respectively; configuring
the catalog configuration is just a matter of setting the property via <catalog_name>.<property>. I also don't think we need to worry about conflicts in property names across different catalogs; in the end we use reflection to load the catalog and it's up to the catalog implementation to respect whichever properties we want, so we just need to make sure the properties are well defined.
The docs seemed pretty clear to me about the original intention of the "jdbc" prefix for the JDBC catalog https://iceberg.apache.org/docs/nightly/jdbc/?h=jdbc#configurations is that it's specifically meant for configuring JDBC connections (username/port/db name that are in the URL itself).
`retryable_status_codes` is *not* configuring the JDBC connection URL.
It is a catalog level property which controls when the catalog should retry establishing an entirely new connection so that's why I think it should be namespaced as it's own property (same as any other property that we do for other catalogs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1588420304
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -21,17 +21,39 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTransientException;
+import java.util.Arrays;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.ClientPoolImpl;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
public class JdbcClientPool extends ClientPoolImpl<Connection, SQLException> {
+ /**
+ * The following are common retryable SQLSTATE error codes which are generic across vendors.
+ *
+ * <ul>
+ * <li>08000: Generic Connection Exception
+ * <li>08003: Connection does not exist
+ * <li>08006: Connection failure
+ * <li>08007: Transaction resolution unknown
+ * </ul>
+ *
+ * See https://en.wikipedia.org/wiki/SQLSTATE for more details.
+ */
+ static final Set<String> COMMON_RETRYABLE_CONNECTION_SQL_STATES =
Review Comment:
Sure it seems like 40001 is typically a transaction rollback prompted by a database's deadlock detector kicking in and essentially killing one of the operations.
I think I'll add 40001 since it can be safely retried and is a common status code across vendors.
I'm a bit more hesitant to add database vendor specific codes (like 40P01) to this default set, since it sets a precedent to be a dumping ground for many states. I'd advocate (at least for now) we keep this default set to be a common set, and users pass in custom codes for other cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565848626
##########
core/src/main/java/org/apache/iceberg/ClientPoolImpl.java:
##########
@@ -56,26 +67,36 @@ public <R> R run(Action<R, C, E> action, boolean retry) throws E, InterruptedExc
C client = get();
try {
return action.run(client);
-
} catch (Exception exc) {
- if (retry && isConnectionException(exc)) {
- try {
- client = reconnect(client);
- } catch (Exception ignored) {
- // if reconnection throws any exception, rethrow the original failure
- throw reconnectExc.cast(exc);
- }
-
- return action.run(client);
+ if (!isConnectionException(exc)) {
+ throw exc;
}
- throw exc;
+ return retryAction(action, exc, client);
} finally {
release(client);
}
}
+ private <R> R retryAction(Action<R, C, E> action, Exception originalFailure, C client) throws E {
+ int retryAttempts = 0;
+ while (retryAttempts < maxRetries) {
+ try {
+ C reconnectedClient = reconnect(client);
+ return action.run(reconnectedClient);
+ } catch (Exception exc) {
+ if (isConnectionException(exc)) {
+ retryAttempts++;
+ } else {
+ throw reconnectExc.cast(originalFailure);
+ }
+ }
+ }
+
+ throw reconnectExc.cast(originalFailure);
Review Comment:
So my thoughts @jbonofre:
1.) C3P0 is a great pooling library! Less familiar with DBCP2 but I'm sure it's in a similar vein. We should leverage such libraries where we can to avoid reimplementing this sort of complex, redundant logic.
2.) That being said, given that long term we would be separating out the catalog into a separate project from what we discussed in the past community sync, the question comes down to (I think) :
do we want to do the work of adding the C3P0 dependency, changing the code to work with that at this point in time, or do we want to do it after a migration to a separate project?
I fall in the latter camp since then we'd need to maintain the implementation until the migration were to happen; and we've already reached a point where it's (imo) a lot for the Iceberg project itself to maintain. The improvements being worked on for multiple backends/vendors just working is great, since that's what people expect from JDBC but again I'm not sure if we want to that upfront here or just a separate catalog project which we would do just prior to a 2.0 version of Iceberg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565856298
##########
core/src/main/java/org/apache/iceberg/ClientPoolImpl.java:
##########
@@ -56,26 +67,36 @@ public <R> R run(Action<R, C, E> action, boolean retry) throws E, InterruptedExc
C client = get();
try {
return action.run(client);
-
} catch (Exception exc) {
- if (retry && isConnectionException(exc)) {
- try {
- client = reconnect(client);
- } catch (Exception ignored) {
- // if reconnection throws any exception, rethrow the original failure
- throw reconnectExc.cast(exc);
- }
-
- return action.run(client);
+ if (!isConnectionException(exc)) {
+ throw exc;
}
- throw exc;
+ return retryAction(action, exc, client);
} finally {
release(client);
}
}
+ private <R> R retryAction(Action<R, C, E> action, Exception originalFailure, C client) throws E {
+ int retryAttempts = 0;
+ while (retryAttempts < maxRetries) {
+ try {
+ C reconnectedClient = reconnect(client);
+ return action.run(reconnectedClient);
+ } catch (Exception exc) {
+ if (isConnectionException(exc)) {
+ retryAttempts++;
+ } else {
+ throw reconnectExc.cast(originalFailure);
+ }
+ }
+ }
+
+ throw reconnectExc.cast(originalFailure);
Review Comment:
Thanks for the update. I started to use DBCP2 while preparing the refactoring to support multiple vendors/backends. I agree with your approach as a kind of "temporary" improvement, waiting the refactoring.
I was wondering, that makes sense.
I'm ready to push/move forward with the "new" JDBC Catalog 😄
Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1597014821
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java:
##########
@@ -40,6 +40,8 @@ final class JdbcUtil {
// property to control if view support is added to the existing database
static final String SCHEMA_VERSION_PROPERTY = JdbcCatalog.PROPERTY_PREFIX + "schema-version";
+ static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
>But it also means that strict-mode and schema-version should not have been prefixed by jdbc then.
Right, I think they should not have been.
>We can change that, but as it's user facing, it could be problematic.
> I can work on a PR to deprecated jdbc prefix (not still supporting it) for strict-mode and schema-version
Sure, we can either deprecate the prefix or just leave as is. Since it's relatively early days for strict-mode/schema-version in the catalog (e.g. it'll practically take time for these paths to get exercised, as views get adopted) I think it makes most sense just to deprecate and correct it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1565013391
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -43,8 +53,16 @@ public JdbcClientPool(String dbUrl, Map<String, String> props) {
}
public JdbcClientPool(int poolSize, String dbUrl, Map<String, String> props) {
- super(poolSize, SQLNonTransientConnectionException.class, true);
+ super(poolSize, SQLTransientException.class, true);
properties = props;
+ retryableStatusCodes = Sets.newHashSet();
+ retryableStatusCodes.addAll(COMMON_RETRYABLE_CONNECTION_SQL_STATES);
+ String configuredRetryableStatuses = props.get(JdbcCatalog.RETRYABLE_STATUS_CODES);
+ if (configuredRetryableStatuses != null) {
Review Comment:
I'm presenting this configuration option since at the moment I believe it's the most practical way for people to work around production issues they hit today.
Long term, if we separate JDBC catalog into a separate library (I forgot where we landed on where JDBC fits in in terms of the catalog discussion that's happened in the past) that library could define status codes for each common, industry-standard database so a user doesn't have to specify anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581117582
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -21,17 +21,27 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTransientException;
+import java.util.Arrays;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.ClientPoolImpl;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
public class JdbcClientPool extends ClientPoolImpl<Connection, SQLException> {
private final String dbUrl;
private final Map<String, String> properties;
+ private Set<String> retryableStatusCodes;
+
+ private static final Set<String> COMMON_RETRYABLE_CONNECTION_SQL_STATES =
Review Comment:
I think it would be helpful to add some javadoc and document what those errors actually mean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581123625
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java:
##########
@@ -40,6 +40,8 @@ final class JdbcUtil {
// property to control if view support is added to the existing database
static final String SCHEMA_VERSION_PROPERTY = JdbcCatalog.PROPERTY_PREFIX + "schema-version";
+ static final String RETRYABLE_STATUS_CODES = "retryable_status_codes";
Review Comment:
should this have `JdbcCatalog.PROPERTY_PREFIX` as a prefix like the other properties have?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581127109
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ class RetryableException extends RuntimeException {}
+
+ class NonRetryableException extends RuntimeException {}
+
+ class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
+ public void testRetrySucceedsWithinMaxAttempts() {
+ int maxRetries = 5;
+ int succeedAfterAttempts = 3;
+ try (MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries)) {
+ int actions = mockClientPool.run(client -> client.succeedAfter(succeedAfterAttempts));
+ Assertions.assertThat(actions)
+ .as("There should be exactly one successful action invocation")
+ .isEqualTo(1);
+ Assertions.assertThat(mockClientPool.reconnectionAttempts())
+ .isEqualTo(succeedAfterAttempts - 1);
+ } catch (Exception e) {
Review Comment:
I think it's fine to just add `throws ...` to the test method instead of catching/re-throwing stuff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1581851465
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ class RetryableException extends RuntimeException {}
+
+ class NonRetryableException extends RuntimeException {}
+
+ class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
+ public void testRetrySucceedsWithinMaxAttempts() {
+ int maxRetries = 5;
+ int succeedAfterAttempts = 3;
+ try (MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries)) {
+ int actions = mockClientPool.run(client -> client.succeedAfter(succeedAfterAttempts));
+ Assertions.assertThat(actions)
+ .as("There should be exactly one successful action invocation")
+ .isEqualTo(1);
+ Assertions.assertThat(mockClientPool.reconnectionAttempts())
+ .isEqualTo(succeedAfterAttempts - 1);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Test
+ public void testRetriesExhaustedAndSurfacesFailure() {
+ int maxRetries = 3;
+ int succeedAfterAttempts = 5;
+ MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries);
+ assertThrows(
Review Comment:
Yes, fixed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1582743700
##########
core/src/test/java/org/apache/iceberg/TestClientPoolImpl.java:
##########
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+import org.junit.jupiter.api.Test;
+
+public class TestClientPoolImpl {
+
+ static class RetryableException extends RuntimeException {}
+
+ static class NonRetryableException extends RuntimeException {}
+
+ static class MockClient {
+ boolean closed = false;
+
+ int actions = 0;
+
+ int retryableFailures = 0;
+
+ public void close() {
+ closed = true;
+ }
+
+ public int successfulAction() {
+ actions++;
+ return actions;
+ }
+
+ int succeedAfter(int succeedAfterAttempts) {
+ if (retryableFailures == succeedAfterAttempts - 1) {
+ return successfulAction();
+ }
+
+ retryableFailures++;
+ throw new RetryableException();
+ }
+
+ int failWithNonRetryable() {
+ throw new NonRetryableException();
+ }
+ }
+
+ static class MockClientPoolImpl extends ClientPoolImpl<MockClient, Exception> {
+
+ private int reconnectionAttempts;
+
+ MockClientPoolImpl(
+ int poolSize,
+ Class<? extends Exception> reconnectExc,
+ boolean retryByDefault,
+ int numRetries) {
+ super(poolSize, reconnectExc, retryByDefault, numRetries);
+ }
+
+ @Override
+ protected MockClient newClient() {
+ return new MockClient();
+ }
+
+ @Override
+ protected MockClient reconnect(MockClient client) {
+ reconnectionAttempts++;
+ return client;
+ }
+
+ @Override
+ protected void close(MockClient client) {
+ client.close();
+ }
+
+ int reconnectionAttempts() {
+ return reconnectionAttempts;
+ }
+ }
+
+ @Test
+ public void testRetrySucceedsWithinMaxAttempts() throws Exception {
+ int maxRetries = 5;
+ int succeedAfterAttempts = 3;
+ try (MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries)) {
+ int actions = mockClientPool.run(client -> client.succeedAfter(succeedAfterAttempts));
+ assertThat(actions)
+ .as("There should be exactly one successful action invocation")
+ .isEqualTo(1);
+ assertThat(mockClientPool.reconnectionAttempts()).isEqualTo(succeedAfterAttempts - 1);
+ }
+ }
+
+ @Test
+ public void testRetriesExhaustedAndSurfacesFailure() {
+ int maxRetries = 3;
+ int succeedAfterAttempts = 5;
+ MockClientPoolImpl mockClientPool =
+ new MockClientPoolImpl(2, RetryableException.class, true, maxRetries);
+ assertThatThrownBy(
+ () -> {
Review Comment:
nit: {} can be removed here and in the other tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1588420304
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -21,17 +21,39 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTransientException;
+import java.util.Arrays;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.ClientPoolImpl;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
public class JdbcClientPool extends ClientPoolImpl<Connection, SQLException> {
+ /**
+ * The following are common retryable SQLSTATE error codes which are generic across vendors.
+ *
+ * <ul>
+ * <li>08000: Generic Connection Exception
+ * <li>08003: Connection does not exist
+ * <li>08006: Connection failure
+ * <li>08007: Transaction resolution unknown
+ * </ul>
+ *
+ * See https://en.wikipedia.org/wiki/SQLSTATE for more details.
+ */
+ static final Set<String> COMMON_RETRYABLE_CONNECTION_SQL_STATES =
Review Comment:
Sure it seems like 40001 is typically a transaction rollback prompted by a database's deadlock detector kicking in and essentially killing one of the operations.
I think I'll add 40001 since it can be safely retried and is a common status code across vendors.
I'm a bit more hesitant to add database vendor specific codes (like 40P01) to this default set, since it sets a precedent to be a dumping ground for many states. I'd advocate we keep this default set to be a common set, and users pass in custom codes for other cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Retry connections in JDBC catalog with user configured error code list [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #10140:
URL: https://github.com/apache/iceberg/pull/10140#discussion_r1589246926
##########
core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java:
##########
@@ -21,17 +21,39 @@
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
-import java.sql.SQLNonTransientConnectionException;
+import java.sql.SQLTransientException;
+import java.util.Arrays;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.ClientPoolImpl;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
public class JdbcClientPool extends ClientPoolImpl<Connection, SQLException> {
+ /**
+ * The following are common retryable SQLSTATE error codes which are generic across vendors.
+ *
+ * <ul>
+ * <li>08000: Generic Connection Exception
+ * <li>08003: Connection does not exist
+ * <li>08006: Connection failure
+ * <li>08007: Transaction resolution unknown
+ * </ul>
+ *
+ * See https://en.wikipedia.org/wiki/SQLSTATE for more details.
+ */
+ static final Set<String> COMMON_RETRYABLE_CONNECTION_SQL_STATES =
Review Comment:
`40P01` can be provided by the user, if using PostgreSQL. So, OK to add `40001` state only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org