You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "jbonofre (via GitHub)" <gi...@apache.org> on 2026/10/07 13:46:38 UTC

Re: [PR] Avoid deleting table metadata on ambiguous commit failures [polaris]

jbonofre commented on code in PR #5263:
URL: https://github.com/apache/polaris/pull/5263#discussion_r4207497508


##########
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/DatasourceOperations.java:
##########
@@ -244,19 +254,37 @@ public int executeUpdate(QueryGenerator.PreparedQuery preparedQuery) throws SQLE
     return withRetries(
         () -> {
           logQuery(preparedQuery);
-          try (Connection connection = borrowConnection();
+          boolean commitStarted = false;
+          try (Connection connection = acquireConnection();
               PreparedStatement statement = connection.prepareStatement(preparedQuery.sql())) {
             List<Object> params = preparedQuery.parameters();
             for (int i = 0; i < params.size(); i++) {
               statement.setObject(i + 1, params.get(i));
             }
             boolean autoCommit = connection.getAutoCommit();
-            connection.setAutoCommit(true);
+            boolean success = false;
+            // Drive the transaction explicitly so the ambiguity window is the commit() call alone:
+            // any failure before commit leaves the transaction uncommitted (a definite non-write).
+            connection.setAutoCommit(false);
+            int rowsUpdated;
             try {
-              return statement.executeUpdate();
+              rowsUpdated = statement.executeUpdate();
+              success = true;
             } finally {
-              connection.setAutoCommit(autoCommit);
+              try {
+                if (success) {
+                  commitStarted = true;
+                  connection.commit();
+                } else {
+                  connection.rollback();
+                }
+              } finally {
+                connection.setAutoCommit(autoCommit);

Review Comment:
   If `rollback()` throws here (e.g. the connection is already dead), that exception replaces the original `SQLException` from `executeUpdate()`. `classifyByPhase` and the retry logic then see the rollback error instead of the real one. For instance a 23505 unique violation (if I remember correctly), which `persistEntity` handles specifically, would be lost.
   
   Maybe we could keep the original exception and attach the rollback failure as suppressed?
   
   
   ```suggestion
    } else {
      try {
        connection.rollback();
      } catch (SQLException rollbackFailure) {
        // keep the original failure, which is the one callers classify
        failure.addSuppressed(rollbackFailure);
      }
    }
   ```
   
   This needs the original exception captured in a variable, since the `finally` doesn't have it today.



##########
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/DatasourceOperations.java:
##########
@@ -244,19 +254,37 @@ public int executeUpdate(QueryGenerator.PreparedQuery preparedQuery) throws SQLE
     return withRetries(
         () -> {
           logQuery(preparedQuery);
-          try (Connection connection = borrowConnection();
+          boolean commitStarted = false;
+          try (Connection connection = acquireConnection();
               PreparedStatement statement = connection.prepareStatement(preparedQuery.sql())) {
             List<Object> params = preparedQuery.parameters();
             for (int i = 0; i < params.size(); i++) {
               statement.setObject(i + 1, params.get(i));
             }
             boolean autoCommit = connection.getAutoCommit();
-            connection.setAutoCommit(true);
+            boolean success = false;
+            // Drive the transaction explicitly so the ambiguity window is the commit() call alone:
+            // any failure before commit leaves the transaction uncommitted (a definite non-write).
+            connection.setAutoCommit(false);

Review Comment:
   This changes single-statement writes from auto-commit to an explicit transaction plus `commit()`. I understand the intent from the comment, which is to make the ambiguity window just the `commit()` call.
   
   I have two questions:
   - Is the extra round trip acceptable on the hot write path?
   - Is it safe for callers that hand in a connection already inside a transaction? `setAutoCommit(autoCommit)` restores the flag, but `commit()` here would commit that outer work.
   
   Nit: this is user-visible behavior, so it should also be in the `CHANGELOG`.



##########
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/DatasourceOperations.java:
##########
@@ -358,25 +399,131 @@ public Integer execute(Connection connection, QueryGenerator.PreparedQuery prepa
         statement.setObject(i + 1, params.get(i));
       }
       return statement.executeUpdate();
+    } catch (SQLException e) {
+      // This statement runs on a connection whose commit is owned by the enclosing
+      // runWithinTransaction; it never commits here, so any failure is before the commit boundary.
+      throw classifyByPhase(e, false);
+    }
+  }
+
+  /**
+   * Whether a SQLException indicates the statement may already have been applied on the server
+   * while the client cannot confirm the outcome (connection loss, timeout, cancellation).
+   *
+   * <p>Callers must not treat these as definite failures for cleanup or safe retry of CAS writes.
+   */
+  public boolean isAmbiguousCommitOutcome(SQLException e) {
+    // A definite non-write (nothing sent to the server: connection acquisition, statement
+    // preparation, parameter binding, or auto-commit setup failed) is never an ambiguous outcome,
+    // even though the driver may tag it with a connection-class SQLSTATE that would otherwise match
+    // below. Check this first so classification is independent of call order.
+    if (isWriteNotStartedFailure(e)) {
+      return false;
+    }
+    // Walk the cause chain: withRetries rewraps the original SQLException in a generic SQLException
+    // on its final throw, so the subtype/SQLSTATE that identifies an ambiguous outcome (timeout,
+    // connection loss) is frequently only visible on a cause rather than the top-level exception.
+    for (Throwable t = e; t != null; t = t.getCause()) {
+      if (t instanceof SQLException && indicatesAmbiguousOutcome((SQLException) t)) {
+        return true;
+      }
+    }
+    return false;
+  }
+
+  private static boolean indicatesAmbiguousOutcome(SQLException e) {
+    if (e instanceof java.sql.SQLTimeoutException) {
+      return true;
+    }
+    if (e instanceof java.sql.SQLTransientConnectionException
+        || e instanceof java.sql.SQLNonTransientConnectionException) {
+      return true;
+    }
+    String sqlState = e.getSQLState();
+    if (sqlState != null) {
+      if (sqlState.startsWith(CONNECTION_EXCEPTION_SQL_STATE_CLASS)
+          || sqlState.equals(POSTGRES_QUERY_CANCELED_SQL_STATE)
+          || sqlState.equals(ODBC_TIMEOUT_SQL_STATE)) {
+        return true;
+      }
     }
+    return messageContainsAny(
+        e,
+        "connection reset",
+        "connection refused",
+        "connection is closed",
+        "broken pipe",
+        "query canceled",
+        "canceling statement due to statement timeout");
   }
 
-  private boolean isRetryable(SQLException e) {
+  /**
+   * Whether a failure that happened before the mutating call was entered (connection acquisition,
+   * statement preparation, parameter binding, auto-commit setup) is present in the cause chain.
+   * Such a failure is a definite non-write, so callers can distinguish it from an ambiguous commit
+   * outcome after execution.
+   */
+  public boolean isWriteNotStartedFailure(SQLException e) {
+    for (Throwable t = e; t != null; t = t.getCause()) {
+      if (t instanceof WriteNotStartedException) {
+        return true;
+      }
+    }
+    return false;
+  }
+
+  /**
+   * Whether the exception message contains any of the given lowercase needles. The message is
+   * lowercased once; a null message matches nothing.
+   */
+  private static boolean messageContainsAny(SQLException e, String... needles) {
+    String message = e.getMessage();
+    if (message == null) {
+      return false;
+    }
+    String lower = message.toLowerCase(Locale.ROOT);
+    for (String needle : needles) {
+      if (lower.contains(needle)) {
+        return true;
+      }
+    }
+    return false;
+  }
+
+  private boolean isRetryable(SQLException e, boolean mutating) {
+    // A definite non-write (the mutating call was never entered) means the write did not happen, so
+    // retrying the whole operation is always safe, even for writes.
+    if (e instanceof WriteNotStartedException) {
+      return true;
+    }
+    // For mutating operations an ambiguous outcome (connection loss, timeout, cancellation) may
+    // mean the write already committed under auto-commit, so retrying risks a double-apply or a
+    // misreported result. Reads have no commit outcome to protect, so they stay retryable.
+    if (mutating && isAmbiguousCommitOutcome(e)) {

Review Comment:
   Mutating operations with an ambiguous outcome are no longer retried. That seems right to avoid double-apply, and `WriteNotStartedException` still retries. But `testSuccessfulExecutionAfterOneRetry` was changed from a connection reset to 40001.
   
   Nit: can we also have a test that pins the new behavior, where a connection reset on a write is not retried, so the change is explicit?



##########
runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java:
##########
@@ -1704,6 +1707,37 @@ public ViewBuilder withLocation(String newLocation) {
     }
   }
 
+  /**
+   * Whether newly written metadata files may be deleted after a table/view commit failure.
+   *
+   * <p>Follows Iceberg's rule: cleanup is only safe when the commit outcome is a known failure
+   * ({@link CleanableFailure} and related definite-failure types). When the outcome is unknown —
+   * connection loss after the metastore may already have applied the pointer update — the metadata
+   * file must be retained so a published pointer cannot dangle.
+   */
+  @VisibleForTesting
+  static boolean shouldCleanupMetadataOnCommitFailure(Throwable failure) {

Review Comment:
   This is basically an allowlist. Cleanup happens only for `CleanableFailure`, `PersistenceWriteNotStartedException` and a few definite types. Any other `RuntimeException` leaves the new `metadata.json` in place. That is conservative and consistent with Iceberg's unknown-state rule.
   
   The cost is that deterministic non-connection failures, such as a generic SQL error or a deadlock that `JdbcBasePersistenceImpl` wraps in a plain `RuntimeException`, now leak a metadata file on every failed commit. Before this PR, I believe they were deleted.
   
   Was that tradeoff deliberate? If so, Javadoc could be updated. If not, those wrapped failures could be classified as definite failures.



##########
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/DatasourceOperations.java:
##########
@@ -358,25 +399,131 @@ public Integer execute(Connection connection, QueryGenerator.PreparedQuery prepa
         statement.setObject(i + 1, params.get(i));
       }
       return statement.executeUpdate();
+    } catch (SQLException e) {
+      // This statement runs on a connection whose commit is owned by the enclosing
+      // runWithinTransaction; it never commits here, so any failure is before the commit boundary.
+      throw classifyByPhase(e, false);
+    }
+  }
+
+  /**
+   * Whether a SQLException indicates the statement may already have been applied on the server
+   * while the client cannot confirm the outcome (connection loss, timeout, cancellation).
+   *
+   * <p>Callers must not treat these as definite failures for cleanup or safe retry of CAS writes.
+   */
+  public boolean isAmbiguousCommitOutcome(SQLException e) {
+    // A definite non-write (nothing sent to the server: connection acquisition, statement
+    // preparation, parameter binding, or auto-commit setup failed) is never an ambiguous outcome,
+    // even though the driver may tag it with a connection-class SQLSTATE that would otherwise match
+    // below. Check this first so classification is independent of call order.
+    if (isWriteNotStartedFailure(e)) {
+      return false;
+    }
+    // Walk the cause chain: withRetries rewraps the original SQLException in a generic SQLException
+    // on its final throw, so the subtype/SQLSTATE that identifies an ambiguous outcome (timeout,
+    // connection loss) is frequently only visible on a cause rather than the top-level exception.
+    for (Throwable t = e; t != null; t = t.getCause()) {
+      if (t instanceof SQLException && indicatesAmbiguousOutcome((SQLException) t)) {
+        return true;
+      }
+    }
+    return false;
+  }
+
+  private static boolean indicatesAmbiguousOutcome(SQLException e) {

Review Comment:
   Nit: `isAmbiguousCommitOutcome` walks the whole cause chain and matches a message substrings like "broken pipe" and "connection is closed". A connection-like exception anywhere in the chain turns a known failure into an unknown commit state (HTTP 500) and suppresses cleanup. 
   
   Could the substring match be limited to the top-level exception and the commit phase?



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/PersistenceWriteNotStartedException.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.persistence;
+
+import jakarta.ws.rs.core.Response;
+import org.apache.polaris.core.exceptions.PolarisException;
+
+/**
+ * Thrown when a persistence write never started because the backing store could not be reached (for
+ * example, a JDBC connection could not be acquired before any statement was sent). This is the
+ * definite-non-write counterpart to {@link PersistenceCommitStateUnknownException}: the write
+ * certainly did not happen, so callers may safely clean up anything they wrote in anticipation of
+ * the commit (e.g. a newly written table metadata file) and the request is safe to retry.
+ *
+ * <p>Note: the simple class name is surfaced to clients as the {@code type} field of the REST error
+ * payload (see {@code PolarisExceptionMapper}). Renaming this class changes that wire value.
+ */
+public class PersistenceWriteNotStartedException extends PolarisException {
+
+  public PersistenceWriteNotStartedException(String message, Throwable cause) {
+    super(message, cause);
+  }
+
+  public PersistenceWriteNotStartedException(String message) {
+    super(message);
+  }
+
+  @Override
+  public int httpStatusCode() {
+    // The write definitely did not happen and the store was unreachable, so this is a transient,
+    // safely retryable failure rather than an unknown outcome.
+    return Response.Status.SERVICE_UNAVAILABLE.getStatusCode();

Review Comment:
   Nit: it returns 503 but extends `PolarisException` directly, not `PolarisServiceUnavailableException`. Anything that special-cases the existing 503 type won't see it.
   
   Is that intentional?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org