[I] Dangling lock with StorageBasedLockProvider when storage returns HTTP 5xx on lock release [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan opened a new issue, #20027:
URL: https://github.com/apache/hudi/issues/20027

   ### Describe the problem you faced
   
   `StorageBasedLockProvider` can leave a dangling lock file when the storage backend returns an HTTP 5xx on the lock-expire write.
   
   Observed in production: a writer acquired the lock, completed everything it needed to do inside the lock, and then hit `503 SERVICE_UNAVAILABLE` when releasing. The lock file was left un-expired, and ~30s later a second writer detected the dangling lock and alerted.
   
   This is the same class of problem as #18438 (fixed in #18439 for HTTP 429), but 5xx was never covered.
   
   **Root cause.** All three storage lock clients classify HTTP 5xx as `UNKNOWN_ERROR` — the 5xx branch logs a warning and then falls through to the `UNKNOWN_ERROR` return:
   
   ```java
   } else if (e.getCode() == RATE_LIMIT_ERROR_CODE) {          // 429 -> THROTTLED (fixed in #18439)
     return Pair.of(LockUpsertResult.THROTTLED, Option.empty());
   } else if (e.getCode() >= INTERNAL_SERVER_ERROR_CODE_MIN) { // 5xx -> falls through
     logger.warn(...);
   }
   return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());   // <- 503 lands here
   ```
   
   `tryExpireCurrentLock` maps `UNKNOWN_ERROR` to `ExpireLockResult.FAILED`, and the retry loop in `unlock()` only engages while the result is `THROTTLED`:
   
   ```java
   for (int attempt = 1; attempt <= THROTTLE_MAX_RETRIES && expireResult == ExpireLockResult.THROTTLED; attempt++) {
   ```
   
   So a 503 fails the release on the **first** attempt with no retry at all.
   
   The cloud SDKs do not cover for this either — both clients are deliberately constructed with retries disabled (`S3StorageLockClient` uses `retryStrategy(r -> r.maxAttempts(1))`, `GCSStorageLockClient` uses `RetrySettings.setMaxAttempts(1)`), so a single transient 5xx dangles the lock every time.
   
   Affects `S3StorageLockClient`, `GCSStorageLockClient` and `AzureStorageLockClient`.
   
   ### To Reproduce
   
   1. Configure `StorageBasedLockProvider` against S3, GCS or Azure.
   2. Acquire the lock and do some work.
   3. Have the storage backend return HTTP 503 (or any 5xx) for the lock-expire `PUT`.
   4. `unlock()` throws `FAILED_TO_RELEASE` immediately, with no retry, and the lock file remains un-expired until its lease elapses.
   
   ### Expected behavior
   
   A 5xx should be treated as a retriable, server-side rejection rather than an indeterminate lock state. The conditional write is known not to have taken effect, so retrying the identical write is safe: if the write never landed the precondition still matches and the retry succeeds; if it silently did land the token no longer matches, yielding `412` -> `ACQUIRED_BY_OTHERS`, which is already handled as released. Either way the retry converges.
   
   ### Environment Description
   
   * Hudi version: master
   * Storage: S3 / GCS / Azure Blob
   * Running on Docker? : no
   
   ### Additional context
   
   A fix is ready and will be linked shortly. It adds `LockUpsertResult.TRANSIENT_ERROR` for HTTP 5xx and reuses the existing retry machinery from #18439 on the release path, plus bounded retries on the acquire and renew paths.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org