You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "chrevanthreddy (via GitHub)" <gi...@apache.org> on 2026/01/19 16:49:28 UTC
[PR] Add hudi azure based storage lock [hudi]
chrevanthreddy opened a new pull request, #17951:
URL: https://github.com/apache/hudi/pull/17951
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3885960474
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24d84063a6e9887312aedb0d1372d28c243a20f5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r2825013945
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
Review Comment:
Could we add these new configs as `ConfigProperty` in `AzureStorageLockConfig` extending `HoodieConfig` so that they are surfaced in the docs?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
+ } catch (BlobStorageException e) {
+ // Blob can change/disappear after properties call; treat 404 during stream read as UNKNOWN_ERROR.
+ return Pair.of(handleGetStorageException(e, true), Option.empty());
+ }
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e, false), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e, boolean ignore404) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ if (ignore404) {
+ logger.info("OwnerId: {}, Azure stream read failure detected: {}", ownerId, lockFileUri);
+ } else {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ }
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private @Nonnull Pair<LockGetResult, Option<StorageLockFile>> getLockFileFromBlob(BlobClient blobClient, String eTag) {
+ try (InputStream inputStream = blobClient.openInputStream()) {
+ return Pair.of(LockGetResult.SUCCESS, Option.of(StorageLockFile.createFromStream(inputStream, eTag)));
+ } catch (IOException ioe) {
+ throw new UncheckedIOException("Failed reading blob: " + lockFileUri, ioe);
+ }
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getValue() != null ? response.getValue().getETag() : null;
+ if (newEtag == null) {
+ newEtag = lockBlobClient.getProperties().getETag();
+ }
+ return new StorageLockFile(lockData, newEtag);
+ }
+
+ private LockUpsertResult handleUpsertBlobStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == PRECONDITION_FAILURE_ERROR_CODE || e.getErrorCode() == BlobErrorCode.CONDITION_NOT_MET) {
+ logger.info("OwnerId: {}, Unable to write new lock file. Another process has modified this lockfile {} already.",
+ ownerId, lockFileUri);
+ return LockUpsertResult.ACQUIRED_BY_OTHERS;
+ } else if (code == CONFLICT_ERROR_CODE) {
+ logger.info("OwnerId: {}, Retriable conditional request conflict error: {}", ownerId, lockFileUri);
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ logger.warn("OwnerId: {}, Error writing lock file: {}", ownerId, lockFileUri, e);
+ }
+ return LockUpsertResult.UNKNOWN_ERROR;
+ }
+ @Override
+ public Option<String> readObject(String filePath, boolean checkExistsFirst) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ AzureLocation lockLocation = parseAzureLocation(lockFileUri);
Review Comment:
The lock file URI has already been parsed in the constructor. Could we reuse that instead of reparsing?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
Review Comment:
Could you revise and reuse `ConfigUtils#getLongWithAltKeys` and remove this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3953024359
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df7054b59932d78d931e443bc3ecbd1e4b9b83b4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4252852969
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 637a857b02bfb874dcc5eb5bfec909f94d271326 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932)
* 2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r2819463332
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
Review Comment:
Could `parseAzureLocation` takes the `props` so there is no need to do `location.withProperties(props)` in the next line for simplicity?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
+ } catch (BlobStorageException e) {
+ // Blob can change/disappear after properties call; treat 404 during stream read as UNKNOWN_ERROR.
+ return Pair.of(handleGetStorageException(e, true), Option.empty());
+ }
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e, false), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e, boolean ignore404) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ if (ignore404) {
+ logger.info("OwnerId: {}, Azure stream read failure detected: {}", ownerId, lockFileUri);
+ } else {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ }
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private @Nonnull Pair<LockGetResult, Option<StorageLockFile>> getLockFileFromBlob(BlobClient blobClient, String eTag) {
+ try (InputStream inputStream = blobClient.openInputStream()) {
+ return Pair.of(LockGetResult.SUCCESS, Option.of(StorageLockFile.createFromStream(inputStream, eTag)));
+ } catch (IOException ioe) {
+ throw new UncheckedIOException("Failed reading blob: " + lockFileUri, ioe);
+ }
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getValue() != null ? response.getValue().getETag() : null;
+ if (newEtag == null) {
+ newEtag = lockBlobClient.getProperties().getETag();
+ }
+ return new StorageLockFile(lockData, newEtag);
+ }
+
+ private LockUpsertResult handleUpsertBlobStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == PRECONDITION_FAILURE_ERROR_CODE || e.getErrorCode() == BlobErrorCode.CONDITION_NOT_MET) {
+ logger.info("OwnerId: {}, Unable to write new lock file. Another process has modified this lockfile {} already.",
+ ownerId, lockFileUri);
+ return LockUpsertResult.ACQUIRED_BY_OTHERS;
+ } else if (code == CONFLICT_ERROR_CODE) {
+ logger.info("OwnerId: {}, Retriable conditional request conflict error: {}", ownerId, lockFileUri);
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ logger.warn("OwnerId: {}, Error writing lock file: {}", ownerId, lockFileUri, e);
+ }
+ return LockUpsertResult.UNKNOWN_ERROR;
+ }
+ @Override
+ public Option<String> readObject(String filePath, boolean checkExistsFirst) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ AzureLocation lockLocation = parseAzureLocation(lockFileUri);
+ BlobServiceClient svc = location.blobEndpoint.equals(lockLocation.blobEndpoint)
Review Comment:
Both `readObject` and `writeObject` call `parseAzureLocation` and potentially `createDefaultBlobServiceClient()` on every invocation. If these are called frequently (e.g., audit logging), this creates a new BlobServiceClient each time for non-matching endpoints. Have you considered caching the secondary service client?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
Review Comment:
`readCurrentLockFile` makes two round-trips to Azure: first `getProperties()` to get the ETag, then `openInputStream()` to read the content. The S3 and GCS implementations both do this in a single call. Since the blob content can change between these two calls, it could end up returning an ETag that doesn't match the body, causing inconsistency. Could you use `downloadContentWithResponse()` with ETag extraction from the response headers in a single call?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
Review Comment:
Rename these classes to have `AzureStorage` prefix instead of the `ADLS` to be descriptive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3961436233
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d956dc0beb807d556d51316f42d4728e41be82e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4271792354
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
}, {
"hash" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347",
"triggerID" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266)
* 2992185d3b874b02c9c4ec77e6f96c8ee83770d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4025351733
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 828646839bbe115c06e8a01e982e62ac4fc33292 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3773882206
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215",
"triggerID" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"triggerType" : "PUSH"
}, {
"hash" : "318978795585cc0c8a52b5b367cb7fffd81ee949",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "318978795585cc0c8a52b5b367cb7fffd81ee949",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 318978795585cc0c8a52b5b367cb7fffd81ee949 UNKNOWN
* 43b35a0a65113c81463a237015cb99a18515d352 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4272183102
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
}, {
"hash" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347",
"triggerID" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"triggerType" : "PUSH"
}, {
"hash" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350",
"triggerID" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2992185d3b874b02c9c4ec77e6f96c8ee83770d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347)
* 4107d19f79f42fa5e9c587e0e78936b7d8ca0ace Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add Azure-based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #17951:
URL: https://github.com/apache/hudi/pull/17951
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3961197501
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df7054b59932d78d931e443bc3ecbd1e4b9b83b4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280)
* 2d956dc0beb807d556d51316f42d4728e41be82e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4148800864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 598acc412f2dcbde52a06add8e314d67f80f3ee0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4157673403
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 598acc412f2dcbde52a06add8e314d67f80f3ee0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899)
* 637a857b02bfb874dcc5eb5bfec909f94d271326 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3916791755
@chrevanthreddy I will queue this up for review..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3769357284
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215",
"triggerID" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38e4d44e679d3c93a99bb5300834dc54d1b56c3e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4272270816
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
}, {
"hash" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347",
"triggerID" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"triggerType" : "PUSH"
}, {
"hash" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350",
"triggerID" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4107d19f79f42fa5e9c587e0e78936b7d8ca0ace Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3773417227
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215",
"triggerID" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"triggerType" : "PUSH"
}, {
"hash" : "318978795585cc0c8a52b5b367cb7fffd81ee949",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "318978795585cc0c8a52b5b367cb7fffd81ee949",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38e4d44e679d3c93a99bb5300834dc54d1b56c3e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215)
* 318978795585cc0c8a52b5b367cb7fffd81ee949 UNKNOWN
* 43b35a0a65113c81463a237015cb99a18515d352 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3876234540
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43b35a0a65113c81463a237015cb99a18515d352 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "chrevanthreddy (via GitHub)" <gi...@apache.org>.
chrevanthreddy commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r2854671666
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
Review Comment:
These classes are renamed to AzureStorage. Kept these for preserving comments. Will remove after comments are resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3952668758
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24d84063a6e9887312aedb0d1372d28c243a20f5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938)
* df7054b59932d78d931e443bc3ecbd1e4b9b83b4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3769345924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38e4d44e679d3c93a99bb5300834dc54d1b56c3e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4025000037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d956dc0beb807d556d51316f42d4728e41be82e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295)
* 828646839bbe115c06e8a01e982e62ac4fc33292 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4272498623
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
}, {
"hash" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347",
"triggerID" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"triggerType" : "PUSH"
}, {
"hash" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350",
"triggerID" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"triggerType" : "PUSH"
}, {
"hash" : "d2d7cb2002a639a6aeedac2f31c35904fc45b2a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2d7cb2002a639a6aeedac2f31c35904fc45b2a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4107d19f79f42fa5e9c587e0e78936b7d8ca0ace Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350)
* d2d7cb2002a639a6aeedac2f31c35904fc45b2a8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4271782757
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
}, {
"hash" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266)
* 2992185d3b874b02c9c4ec77e6f96c8ee83770d3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r3103708873
##########
hudi-azure/pom.xml:
##########
@@ -0,0 +1,240 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.2.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <modelVersion>4.0.0</modelVersion>
+ <artifactId>hudi-azure</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <!-- Azure SDK versions are intentionally pinned here to keep this module self-contained. -->
+ <azure.storage.blob.version>12.26.0</azure.storage.blob.version>
+ <azure.identity.version>1.12.2</azure.identity.version>
+ <!-- Keep Jackson consistent for Azure SDK integration tests (Hudi core pins older Jackson versions). -->
+ <azure.jackson.version>2.13.5</azure.jackson.version>
Review Comment:
This is to be aligned with existing Jackson version used by Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "chrevanthreddy (via GitHub)" <gi...@apache.org>.
chrevanthreddy commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r2854755359
##########
packaging/hudi-azure-bundle/pom.xml:
##########
@@ -0,0 +1,179 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.2.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>hudi-azure-bundle</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <checkstyle.skip>true</checkstyle.skip>
+ <main.basedir>${project.parent.basedir}</main.basedir>
+ <skipTests>true</skipTests>
+ </properties>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>${maven-shade-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
Review Comment:
yes. All the testing is done on the bundle for Spark and Flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4253612334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3773401161
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215",
"triggerID" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"triggerType" : "PUSH"
}, {
"hash" : "318978795585cc0c8a52b5b367cb7fffd81ee949",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "318978795585cc0c8a52b5b367cb7fffd81ee949",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38e4d44e679d3c93a99bb5300834dc54d1b56c3e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215)
* 318978795585cc0c8a52b5b367cb7fffd81ee949 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r2825430833
##########
packaging/hudi-azure-bundle/pom.xml:
##########
@@ -0,0 +1,179 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.2.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>hudi-azure-bundle</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <checkstyle.skip>true</checkstyle.skip>
+ <main.basedir>${project.parent.basedir}</main.basedir>
+ <skipTests>true</skipTests>
+ </properties>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>${maven-shade-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
Review Comment:
Have you tested the generated Hudi Azure bundle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3885474129
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43b35a0a65113c81463a237015cb99a18515d352 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857)
* 24d84063a6e9887312aedb0d1372d28c243a20f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3961183739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* df7054b59932d78d931e443bc3ecbd1e4b9b83b4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280)
* 2d956dc0beb807d556d51316f42d4728e41be82e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r2912893657
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+
+ // 2. SAS token.
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ // 3. TokenCredential — MI, service principal, or DefaultAzureCredential fallback.
+ return builder.credential(AzureCredentialFactory.getAzureCredential(props)).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
Review Comment:
If the response headers are missing or don't contain `ETag`, `eTag` will be null and get silently passed into the `StorageLockFile`. A subsequent `tryUpsertLockFile` with that file would then treat it as a create (`If-None-Match: *`) instead of an update (`If-Match: <etag>`), which would fail with a precondition error and prevent the rightful owner from renewing the lock. Could you add a null/empty check here similar to what `createOrUpdateLockFileInternal` does (throwing `HoodieLockException`)?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/credentials/AzureCredentialFactory.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.credentials;
+
+import org.apache.hudi.config.AzureStorageLockConfig;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.ClientSecretCredentialBuilder;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.identity.ManagedIdentityCredentialBuilder;
+
+import java.util.Properties;
+
+/**
+ * Factory for resolving an Azure {@link TokenCredential} from Hudi properties.
+ *
+ * <p>Credential precedence:
+ * <ol>
+ * <li>User-assigned managed identity ({@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID})
+ * — uses {@code ManagedIdentityCredential}</li>
+ * <li>Service principal ({@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET})
+ * — uses {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — (system-assigned MI,
+ * workload identity, env-var SP, Azure CLI, etc.); suitable for dev and environments
+ * where auth is controlled externally</li>
+ * </ol>
+ *
+ * <p>Note: connection string and SAS token auth are not {@link TokenCredential}-based and are
+ * handled directly by the caller before this factory is consulted.
+ */
+public class AzureCredentialFactory {
+
+ // Shared instance so the token cache and IMDS probe are reused across all clients
+ // that fall through to the default chain.
+ private static final TokenCredential DEFAULT_AZURE_CREDENTIAL =
+ new DefaultAzureCredentialBuilder().build();
+
Review Comment:
This eagerly builds a `DefaultAzureCredential` at class-load time, which can trigger IMDS endpoint probes and network calls. In non-Azure environments (local dev, on-prem Spark), this could cause unexpected hangs or timeouts when the class is first loaded — even if the user configured a connection string or SAS token. Have you considered lazy initialization (e.g., a holder pattern or `synchronized` lazy init)?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
+ } catch (BlobStorageException e) {
+ // Blob can change/disappear after properties call; treat 404 during stream read as UNKNOWN_ERROR.
+ return Pair.of(handleGetStorageException(e, true), Option.empty());
+ }
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e, false), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e, boolean ignore404) {
Review Comment:
The fallback `lockBlobClient.getProperties().getETag()` introduces a TOCTOU race — another process could update the blob between the upload response and this getProperties call, returning a stale or wrong ETag. The newer `AzureStorageLockClient` avoids this by reading from response headers. Since this class is going away, just flagging for awareness.
##########
hudi-azure/src/main/java/org/apache/hudi/config/AzureStorageLockConfig.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.config.LockConfiguration;
+
+/**
+ * Hoodie Configs for Azure based storage locks.
+ */
+@ConfigClassProperty(name = "Azure based Locks Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ subGroupName = ConfigGroups.SubGroupNames.LOCK,
+ description = "Configs that control Azure Blob/ADLS based locking mechanisms "
+ + "required for concurrency control between writers to a Hudi table.")
+public class AzureStorageLockConfig extends HoodieConfig {
+
+ private static final String AZURE_BASED_LOCK_PROPERTY_PREFIX = LockConfiguration.LOCK_PREFIX + "azure.";
+
+ public static final ConfigProperty<String> AZURE_CONNECTION_STRING = ConfigProperty
+ .key(AZURE_BASED_LOCK_PROPERTY_PREFIX + "connection.string")
+ .noDefaultValue()
+ .markAdvanced()
Review Comment:
Let's add `.sinceVersion("1.2.0")` to all these new configs
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
Review Comment:
Use `ConfigUtils.getStringWithAltKeys` and no need to create `AZURE_CONNECTION_STRING`
##########
pom.xml:
##########
@@ -112,7 +114,7 @@
<confluent.version>5.5.0</confluent.version>
<glassfish.version>2.17</glassfish.version>
<glassfish.el.version>3.0.1-b12</glassfish.el.version>
- <parquet.version>1.10.1</parquet.version>
+ <parquet.version>1.13.1</parquet.version>
Review Comment:
+1 @chrevanthreddy let's revert this
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ String eTag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ StorageLockFile lockFile = StorageLockFile.createFromStream(response.getValue().toStream(), eTag);
+ return Pair.of(LockGetResult.SUCCESS, Option.of(lockFile));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ if (newEtag == null && response.getValue() != null) {
+ newEtag = response.getValue().getETag();
+ }
+ if (newEtag == null || newEtag.isEmpty()) {
+ throw new HoodieLockException("Missing ETag in Azure upload response for lock file: " + lockFileUri);
+ }
+ return new StorageLockFile(lockData, newEtag);
+ }
+
+ private LockUpsertResult handleUpsertBlobStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == PRECONDITION_FAILURE_ERROR_CODE || e.getErrorCode() == BlobErrorCode.CONDITION_NOT_MET) {
+ logger.info("OwnerId: {}, Unable to write new lock file. Another process has modified this lockfile {} already.",
+ ownerId, lockFileUri);
+ return LockUpsertResult.ACQUIRED_BY_OTHERS;
+ } else if (code == CONFLICT_ERROR_CODE) {
+ logger.info("OwnerId: {}, Retriable conditional request conflict error: {}", ownerId, lockFileUri);
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ logger.warn("OwnerId: {}, Error writing lock file: {}", ownerId, lockFileUri, e);
Review Comment:
The upper layer will handle the `UNKNOWN_ERROR` including any other transient errors as a whole so I think this is OK.
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ String eTag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
Review Comment:
We should throw an exception if the `ETag` is not returned.
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+
+ // 2. SAS token.
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ // 3. TokenCredential — MI, service principal, or DefaultAzureCredential fallback.
+ return builder.credential(AzureCredentialFactory.getAzureCredential(props)).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ String eTag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ StorageLockFile lockFile = StorageLockFile.createFromStream(response.getValue().toStream(), eTag);
+ return Pair.of(LockGetResult.SUCCESS, Option.of(lockFile));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ if (newEtag == null && response.getValue() != null) {
+ newEtag = response.getValue().getETag();
Review Comment:
AFAIK, `response.getValue().getETag()` returns the unquoted form (`0x8D..`) whereas `response.getHeaders().getValue("ETag")` returns the quoted form (`"0x8D..."`). This creates inconsistency in the stored eTag and causes issue for conditional writes when passing the eTag to `conditions.setIfMatch`. Could you double check the eTag form of both when running on Azure storage and make a fix accordingly?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/credentials/AzureCredentialFactory.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.credentials;
+
+import org.apache.hudi.config.AzureStorageLockConfig;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.ClientSecretCredentialBuilder;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.identity.ManagedIdentityCredentialBuilder;
+
+import java.util.Properties;
+
+/**
+ * Factory for resolving an Azure {@link TokenCredential} from Hudi properties.
+ *
+ * <p>Credential precedence:
+ * <ol>
+ * <li>User-assigned managed identity ({@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID})
+ * — uses {@code ManagedIdentityCredential}</li>
+ * <li>Service principal ({@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET})
+ * — uses {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — (system-assigned MI,
+ * workload identity, env-var SP, Azure CLI, etc.); suitable for dev and environments
+ * where auth is controlled externally</li>
+ * </ol>
+ *
+ * <p>Note: connection string and SAS token auth are not {@link TokenCredential}-based and are
+ * handled directly by the caller before this factory is consulted.
+ */
+public class AzureCredentialFactory {
+
+ // Shared instance so the token cache and IMDS probe are reused across all clients
+ // that fall through to the default chain.
+ private static final TokenCredential DEFAULT_AZURE_CREDENTIAL =
+ new DefaultAzureCredentialBuilder().build();
+
Review Comment:
Similar to #17552 to avoid initialization failures causing class loading issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3952701512
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24d84063a6e9887312aedb0d1372d28c243a20f5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938)
* df7054b59932d78d931e443bc3ecbd1e4b9b83b4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r2852857934
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
Review Comment:
is this class just used by tests?
<img width="606" height="283" alt="Image" src="https://github.com/user-attachments/assets/8be77b55-3157-42d7-bc50-773ca52d9919" />
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ String eTag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
Review Comment:
I am a little nervous with the `eTag` being `null`. Won't we pass the null back as precondition check next time and it ll always succeed.
Should we error out instead?
##########
hudi-azure/src/main/java/org/apache/hudi/config/AzureStorageLockConfig.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.config.LockConfiguration;
+
+/**
+ * Hoodie Configs for Azure based storage locks.
+ */
+@ConfigClassProperty(name = "Azure based Locks Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ subGroupName = ConfigGroups.SubGroupNames.LOCK,
+ description = "Configs that control Azure Blob/ADLS based locking mechanisms "
+ + "required for concurrency control between writers to a Hudi table.")
+public class AzureStorageLockConfig extends HoodieConfig {
+
+ private static final String AZURE_BASED_LOCK_PROPERTY_PREFIX = LockConfiguration.LOCK_PREFIX + "azure.";
+
+ public static final ConfigProperty<String> AZURE_CONNECTION_STRING = ConfigProperty
Review Comment:
same question as above - on whether this is typical azure production setup. per some cursory research, it seems azure calls out some risks/precautions with connection strings? https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string
##########
hudi-azure/src/main/java/org/apache/hudi/config/AzureStorageLockConfig.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.config.LockConfiguration;
+
+/**
+ * Hoodie Configs for Azure based storage locks.
+ */
+@ConfigClassProperty(name = "Azure based Locks Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ subGroupName = ConfigGroups.SubGroupNames.LOCK,
+ description = "Configs that control Azure Blob/ADLS based locking mechanisms "
+ + "required for concurrency control between writers to a Hudi table.")
+public class AzureStorageLockConfig extends HoodieConfig {
+
+ private static final String AZURE_BASED_LOCK_PROPERTY_PREFIX = LockConfiguration.LOCK_PREFIX + "azure.";
+
+ public static final ConfigProperty<String> AZURE_CONNECTION_STRING = ConfigProperty
Review Comment:
if you intend this for dev/testing, lets update docs to specify so.
##########
hudi-azure/src/main/java/org/apache/hudi/config/AzureStorageLockConfig.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.config.LockConfiguration;
+
+/**
+ * Hoodie Configs for Azure based storage locks.
+ */
+@ConfigClassProperty(name = "Azure based Locks Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ subGroupName = ConfigGroups.SubGroupNames.LOCK,
+ description = "Configs that control Azure Blob/ADLS based locking mechanisms "
+ + "required for concurrency control between writers to a Hudi table.")
+public class AzureStorageLockConfig extends HoodieConfig {
+
+ private static final String AZURE_BASED_LOCK_PROPERTY_PREFIX = LockConfiguration.LOCK_PREFIX + "azure.";
+
+ public static final ConfigProperty<String> AZURE_CONNECTION_STRING = ConfigProperty
+ .key(AZURE_BASED_LOCK_PROPERTY_PREFIX + "connection.string")
+ .noDefaultValue()
+ .markAdvanced()
+ .withDocumentation("For Azure based lock provider, optional Azure Storage connection string used "
+ + "for authenticating BlobServiceClient.");
+
+ public static final ConfigProperty<String> AZURE_SAS_TOKEN = ConfigProperty
+ .key(AZURE_BASED_LOCK_PROPERTY_PREFIX + "sas.token")
+ .noDefaultValue()
+ .markAdvanced()
+ .withDocumentation("For Azure based lock provider, optional SAS token used for "
Review Comment:
nit: document the minimum permissions required on the token. Read, Write, Create?
##########
pom.xml:
##########
@@ -112,7 +114,7 @@
<confluent.version>5.5.0</confluent.version>
<glassfish.version>2.17</glassfish.version>
<glassfish.el.version>3.0.1-b12</glassfish.el.version>
- <parquet.version>1.10.1</parquet.version>
+ <parquet.version>1.13.1</parquet.version>
Review Comment:
is this change and the version bump below related to the locking somehow? if not, can we revert.. and decouple from this PR.
##########
hudi-azure/pom.xml:
##########
@@ -0,0 +1,240 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.2.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <modelVersion>4.0.0</modelVersion>
+ <artifactId>hudi-azure</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <!-- Azure SDK versions are intentionally pinned here to keep this module self-contained. -->
+ <azure.storage.blob.version>12.26.0</azure.storage.blob.version>
+ <azure.identity.version>1.12.2</azure.identity.version>
+ <!-- Keep Jackson consistent for Azure SDK integration tests (Hudi core pins older Jackson versions). -->
+ <azure.jackson.version>2.13.5</azure.jackson.version>
+ </properties>
+
+ <dependencies>
+ <!-- Logging -->
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-1.2-api</artifactId>
+ </dependency>
+
+ <!-- Lombok -->
+ <dependency>
+ <groupId>org.projectlombok</groupId>
+ <artifactId>lombok</artifactId>
+ </dependency>
+
+ <!-- Hoodie -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-client-common</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-common</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <!-- Hadoop (needed by shared test utilities / configs, consistent with hudi-gcp) -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Azure SDK (Blob + DefaultAzureCredential) -->
+ <dependency>
+ <groupId>com.azure</groupId>
+ <artifactId>azure-storage-blob</artifactId>
+ <version>${azure.storage.blob.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.azure</groupId>
+ <artifactId>azure-identity</artifactId>
+ <version>${azure.identity.version}</version>
+ </dependency>
+
+ <!-- Test -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-tests-common</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-client-common</artifactId>
+ <version>${project.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-hadoop-common</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ <type>test-jar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-common</artifactId>
+ <version>${project.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-core</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <!-- Azure SDK requires a consistent Jackson stack for XML support; override for tests -->
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.datatype</groupId>
+ <artifactId>jackson-datatype-jsr310</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.dataformat</groupId>
+ <artifactId>jackson-dataformat-xml</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ <version>${testcontainers.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <version>${testcontainers.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.esotericsoftware</groupId>
+ <artifactId>kryo-shaded</artifactId>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <profiles>
+ <profile>
+ <id>azure-integration-tests</id>
Review Comment:
I approved some extra workflows, that showed up. but +1 lets verify
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
Review Comment:
Should we use different credential builder here for typical azure production setups? (not very fam myself there).
Asking since IIRC `HoodieAWSConfig` supports role ARN, access key, secret key, session token etc..
##########
packaging/hudi-azure-bundle/pom.xml:
##########
@@ -0,0 +1,179 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.2.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>hudi-azure-bundle</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <checkstyle.skip>true</checkstyle.skip>
+ <main.basedir>${project.parent.basedir}</main.basedir>
+ <skipTests>true</skipTests>
+ </properties>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.rat</groupId>
+ <artifactId>apache-rat-plugin</artifactId>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>${maven-shade-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
Review Comment:
+1 it would be great if you can add status on that
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ String eTag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ StorageLockFile lockFile = StorageLockFile.createFromStream(response.getValue().toStream(), eTag);
+ return Pair.of(LockGetResult.SUCCESS, Option.of(lockFile));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ if (newEtag == null && response.getValue() != null) {
+ newEtag = response.getValue().getETag();
+ }
+ if (newEtag == null || newEtag.isEmpty()) {
+ throw new HoodieLockException("Missing ETag in Azure upload response for lock file: " + lockFileUri);
+ }
+ return new StorageLockFile(lockData, newEtag);
+ }
+
+ private LockUpsertResult handleUpsertBlobStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == PRECONDITION_FAILURE_ERROR_CODE || e.getErrorCode() == BlobErrorCode.CONDITION_NOT_MET) {
+ logger.info("OwnerId: {}, Unable to write new lock file. Another process has modified this lockfile {} already.",
+ ownerId, lockFileUri);
+ return LockUpsertResult.ACQUIRED_BY_OTHERS;
+ } else if (code == CONFLICT_ERROR_CODE) {
+ logger.info("OwnerId: {}, Retriable conditional request conflict error: {}", ownerId, lockFileUri);
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ logger.warn("OwnerId: {}, Error writing lock file: {}", ownerId, lockFileUri, e);
Review Comment:
Should we `throw e` here instead, since its an UNKNOWN case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4025019070
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d956dc0beb807d556d51316f42d4728e41be82e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295)
* 828646839bbe115c06e8a01e982e62ac4fc33292 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "chrevanthreddy (via GitHub)" <gi...@apache.org>.
chrevanthreddy commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3893757523
@vinothchandar @bhasudha I merged this PR and Alex PR together. Testing looks good to me. Let me know the next steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4148741214
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 828646839bbe115c06e8a01e982e62ac4fc33292 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505)
* 598acc412f2dcbde52a06add8e314d67f80f3ee0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add hudi azure based storage lock [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3787526008
@chrevanthreddy can we fix the PR description.. it ll fail validation.
I approved the workflows to run now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "chrevanthreddy (via GitHub)" <gi...@apache.org>.
chrevanthreddy commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3800356300
Changed title, added description here too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3874774245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43b35a0a65113c81463a237015cb99a18515d352 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r2825426724
##########
hudi-azure/pom.xml:
##########
@@ -0,0 +1,240 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.2.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <modelVersion>4.0.0</modelVersion>
+ <artifactId>hudi-azure</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <!-- Azure SDK versions are intentionally pinned here to keep this module self-contained. -->
+ <azure.storage.blob.version>12.26.0</azure.storage.blob.version>
+ <azure.identity.version>1.12.2</azure.identity.version>
+ <!-- Keep Jackson consistent for Azure SDK integration tests (Hudi core pins older Jackson versions). -->
+ <azure.jackson.version>2.13.5</azure.jackson.version>
+ </properties>
+
+ <dependencies>
+ <!-- Logging -->
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-1.2-api</artifactId>
+ </dependency>
+
+ <!-- Lombok -->
+ <dependency>
+ <groupId>org.projectlombok</groupId>
+ <artifactId>lombok</artifactId>
+ </dependency>
+
+ <!-- Hoodie -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-client-common</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-common</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <!-- Hadoop (needed by shared test utilities / configs, consistent with hudi-gcp) -->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <!-- Azure SDK (Blob + DefaultAzureCredential) -->
+ <dependency>
+ <groupId>com.azure</groupId>
+ <artifactId>azure-storage-blob</artifactId>
+ <version>${azure.storage.blob.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>com.azure</groupId>
+ <artifactId>azure-identity</artifactId>
+ <version>${azure.identity.version}</version>
+ </dependency>
+
+ <!-- Test -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-tests-common</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-client-common</artifactId>
+ <version>${project.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-hadoop-common</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ <type>test-jar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-common</artifactId>
+ <version>${project.version}</version>
+ <classifier>tests</classifier>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-core</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <!-- Azure SDK requires a consistent Jackson stack for XML support; override for tests -->
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.datatype</groupId>
+ <artifactId>jackson-datatype-jsr310</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.dataformat</groupId>
+ <artifactId>jackson-dataformat-xml</artifactId>
+ <version>${azure.jackson.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ <version>${testcontainers.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <version>${testcontainers.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.esotericsoftware</groupId>
+ <artifactId>kryo-shaded</artifactId>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <profiles>
+ <profile>
+ <id>azure-integration-tests</id>
Review Comment:
Could you verify that if the CI run the newly added ITs?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
+ } catch (BlobStorageException e) {
+ // Blob can change/disappear after properties call; treat 404 during stream read as UNKNOWN_ERROR.
+ return Pair.of(handleGetStorageException(e, true), Option.empty());
+ }
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e, false), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e, boolean ignore404) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ if (ignore404) {
+ logger.info("OwnerId: {}, Azure stream read failure detected: {}", ownerId, lockFileUri);
+ } else {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ }
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private @Nonnull Pair<LockGetResult, Option<StorageLockFile>> getLockFileFromBlob(BlobClient blobClient, String eTag) {
+ try (InputStream inputStream = blobClient.openInputStream()) {
+ return Pair.of(LockGetResult.SUCCESS, Option.of(StorageLockFile.createFromStream(inputStream, eTag)));
+ } catch (IOException ioe) {
+ throw new UncheckedIOException("Failed reading blob: " + lockFileUri, ioe);
+ }
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getValue() != null ? response.getValue().getETag() : null;
+ if (newEtag == null) {
+ newEtag = lockBlobClient.getProperties().getETag();
+ }
+ return new StorageLockFile(lockData, newEtag);
+ }
+
+ private LockUpsertResult handleUpsertBlobStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == PRECONDITION_FAILURE_ERROR_CODE || e.getErrorCode() == BlobErrorCode.CONDITION_NOT_MET) {
+ logger.info("OwnerId: {}, Unable to write new lock file. Another process has modified this lockfile {} already.",
+ ownerId, lockFileUri);
+ return LockUpsertResult.ACQUIRED_BY_OTHERS;
+ } else if (code == CONFLICT_ERROR_CODE) {
+ logger.info("OwnerId: {}, Retriable conditional request conflict error: {}", ownerId, lockFileUri);
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ logger.warn("OwnerId: {}, Error writing lock file: {}", ownerId, lockFileUri, e);
+ }
+ return LockUpsertResult.UNKNOWN_ERROR;
+ }
+ @Override
+ public Option<String> readObject(String filePath, boolean checkExistsFirst) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ AzureLocation lockLocation = parseAzureLocation(lockFileUri);
+ BlobServiceClient svc = location.blobEndpoint.equals(lockLocation.blobEndpoint)
+ ? blobServiceClient
+ : createDefaultBlobServiceClient().apply(location.withProperties(clientProperties));
+ BlobClient blobClient = svc.getBlobContainerClient(location.container).getBlobClient(location.blobPath);
+
+ if (checkExistsFirst && !blobClient.exists()) {
+ logger.debug("JSON config file not found: {}", filePath);
+ return Option.empty();
+ }
+ byte[] bytes = blobClient.downloadContent().toBytes();
+ return Option.of(new String(bytes, UTF_8));
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() == NOT_FOUND_ERROR_CODE) {
+ logger.debug("JSON config file not found: {}", filePath);
+ } else {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ }
+ return Option.empty();
+ } catch (Exception e) {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ return Option.empty();
+ }
+ }
+
+ @Override
+ public boolean writeObject(String filePath, String content) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ AzureLocation lockLocation = parseAzureLocation(lockFileUri);
Review Comment:
Similarly, could we avoid this?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
+ } catch (BlobStorageException e) {
+ // Blob can change/disappear after properties call; treat 404 during stream read as UNKNOWN_ERROR.
+ return Pair.of(handleGetStorageException(e, true), Option.empty());
+ }
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e, false), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e, boolean ignore404) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ if (ignore404) {
+ logger.info("OwnerId: {}, Azure stream read failure detected: {}", ownerId, lockFileUri);
+ } else {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ }
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private @Nonnull Pair<LockGetResult, Option<StorageLockFile>> getLockFileFromBlob(BlobClient blobClient, String eTag) {
+ try (InputStream inputStream = blobClient.openInputStream()) {
+ return Pair.of(LockGetResult.SUCCESS, Option.of(StorageLockFile.createFromStream(inputStream, eTag)));
+ } catch (IOException ioe) {
+ throw new UncheckedIOException("Failed reading blob: " + lockFileUri, ioe);
+ }
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getValue() != null ? response.getValue().getETag() : null;
+ if (newEtag == null) {
+ newEtag = lockBlobClient.getProperties().getETag();
+ }
+ return new StorageLockFile(lockData, newEtag);
+ }
+
+ private LockUpsertResult handleUpsertBlobStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == PRECONDITION_FAILURE_ERROR_CODE || e.getErrorCode() == BlobErrorCode.CONDITION_NOT_MET) {
+ logger.info("OwnerId: {}, Unable to write new lock file. Another process has modified this lockfile {} already.",
+ ownerId, lockFileUri);
+ return LockUpsertResult.ACQUIRED_BY_OTHERS;
+ } else if (code == CONFLICT_ERROR_CODE) {
+ logger.info("OwnerId: {}, Retriable conditional request conflict error: {}", ownerId, lockFileUri);
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ logger.warn("OwnerId: {}, Error writing lock file: {}", ownerId, lockFileUri, e);
+ }
+ return LockUpsertResult.UNKNOWN_ERROR;
+ }
+ @Override
+ public Option<String> readObject(String filePath, boolean checkExistsFirst) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ AzureLocation lockLocation = parseAzureLocation(lockFileUri);
+ BlobServiceClient svc = location.blobEndpoint.equals(lockLocation.blobEndpoint)
+ ? blobServiceClient
+ : createDefaultBlobServiceClient().apply(location.withProperties(clientProperties));
+ BlobClient blobClient = svc.getBlobContainerClient(location.container).getBlobClient(location.blobPath);
+
+ if (checkExistsFirst && !blobClient.exists()) {
+ logger.debug("JSON config file not found: {}", filePath);
+ return Option.empty();
+ }
+ byte[] bytes = blobClient.downloadContent().toBytes();
+ return Option.of(new String(bytes, UTF_8));
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() == NOT_FOUND_ERROR_CODE) {
+ logger.debug("JSON config file not found: {}", filePath);
+ } else {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ }
+ return Option.empty();
+ } catch (Exception e) {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ return Option.empty();
+ }
+ }
+
+ @Override
+ public boolean writeObject(String filePath, String content) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ AzureLocation lockLocation = parseAzureLocation(lockFileUri);
+ BlobServiceClient svc = location.blobEndpoint.equals(lockLocation.blobEndpoint)
+ ? blobServiceClient
+ : createDefaultBlobServiceClient().apply(location.withProperties(clientProperties));
+ BlobClient blobClient = svc.getBlobContainerClient(location.container).getBlobClient(location.blobPath);
+ blobClient.upload(BinaryData.fromString(content), true);
+ logger.debug("Successfully wrote object to: {}", filePath);
+ return true;
+ } catch (Exception e) {
+ logger.error("Error writing object to: {}", filePath, e);
+ return false;
+ }
+ }
+
+ @Override
+ public void close() {
+ // BlobServiceClient does not require explicit close. No-op.
+ }
+
+ @VisibleForTesting
+ static AzureLocation parseAzureLocation(String uriString) {
+ try {
+ URI uri = new URI(uriString);
+ String scheme = uri.getScheme();
+ if (scheme == null) {
+ throw new IllegalArgumentException("URI does not contain a valid scheme.");
+ }
+
+ String authority = uri.getAuthority();
+ String path = uri.getPath() == null ? "" : uri.getPath().replaceFirst("/", "");
+
+ // ADLS Gen2: abfs[s]://<container>@<account>.dfs.core.windows.net/<path>
+ if ("abfs".equalsIgnoreCase(scheme) || "abfss".equalsIgnoreCase(scheme)) {
+ if (authority == null || !authority.contains("@")) {
+ throw new IllegalArgumentException("ABFS URI authority must be in the form '<container>@<host>': " + uriString);
+ }
+ String[] parts = authority.split("@", 2);
+ String container = parts[0];
+ String host = parts[1];
+ String endpointHost = dfsHostToBlobHost(host);
+ String endpoint = "https://" + endpointHost;
+ if (container.isEmpty() || path.isEmpty()) {
+ throw new IllegalArgumentException("ABFS URI must contain container and path: " + uriString);
+ }
+ return new AzureLocation(endpoint, container, path, null);
+ }
+
+ // Azure Blob Storage: wasb[s]://<container>@<account>.blob.core.windows.net/<path>
+ if ("wasb".equalsIgnoreCase(scheme) || "wasbs".equalsIgnoreCase(scheme)) {
+ if (authority == null || !authority.contains("@")) {
+ throw new IllegalArgumentException("WASB URI authority must be in the form '<container>@<host>': " + uriString);
+ }
+ String[] parts = authority.split("@", 2);
+ String container = parts[0];
+ String host = parts[1];
+ String endpoint = "https://" + host;
+ if (container.isEmpty() || path.isEmpty()) {
+ throw new IllegalArgumentException("WASB URI must contain container and path: " + uriString);
+ }
+ return new AzureLocation(endpoint, container, path, null);
+ }
+
+ // Direct HTTPS: https://<account>.blob.core.windows.net/<container>/<path>
+ if ("https".equalsIgnoreCase(scheme)) {
+ if (authority == null || authority.isEmpty()) {
+ throw new IllegalArgumentException("HTTPS URI authority missing: " + uriString);
+ }
+ int slash = path.indexOf('/');
+ if (slash <= 0 || slash == path.length() - 1) {
+ throw new IllegalArgumentException("HTTPS URI must be in the form 'https://<host>/<container>/<path>': " + uriString);
+ }
+ String container = path.substring(0, slash);
+ String blobPath = path.substring(slash + 1);
+ return new AzureLocation("https://" + authority, container, blobPath, null);
+ }
+
+ throw new IllegalArgumentException("Unsupported scheme for Azure storage lock: " + scheme
+ + ". Supported schemes: abfs, abfss, wasb, wasbs, https");
+ } catch (URISyntaxException e) {
+ throw new HoodieLockException("Failed to parse Azure URI: " + uriString, e);
+ }
+ }
+
+ private static String dfsHostToBlobHost(String host) {
+ if (host == null) {
+ return null;
+ }
+ if (host.endsWith(".dfs.core.windows.net")) {
+ return host.replace(".dfs.core.windows.net", ".blob.core.windows.net");
+ }
+ return host;
+ }
+
+ @VisibleForTesting
+ static final class AzureLocation {
+ final String blobEndpoint;
+ final String container;
+ final String blobPath;
+ final Properties props;
+
+ AzureLocation(String blobEndpoint, String container, String blobPath, Properties props) {
+ this.blobEndpoint = blobEndpoint;
+ this.container = container;
+ this.blobPath = blobPath;
+ this.props = props;
+ }
Review Comment:
nit: use lombok to auto generate this constructor
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
+ } catch (BlobStorageException e) {
+ // Blob can change/disappear after properties call; treat 404 during stream read as UNKNOWN_ERROR.
+ return Pair.of(handleGetStorageException(e, true), Option.empty());
+ }
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e, false), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e, boolean ignore404) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ if (ignore404) {
+ logger.info("OwnerId: {}, Azure stream read failure detected: {}", ownerId, lockFileUri);
+ } else {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ }
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
Review Comment:
Add logging for `status == CONFLICT_ERROR_CODE`?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
Review Comment:
For other exceptions, like S3 and GCS-based implementation, it should let the exception throw without return.
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
+ } catch (BlobStorageException e) {
+ // Blob can change/disappear after properties call; treat 404 during stream read as UNKNOWN_ERROR.
+ return Pair.of(handleGetStorageException(e, true), Option.empty());
+ }
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e, false), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e, boolean ignore404) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ if (ignore404) {
+ logger.info("OwnerId: {}, Azure stream read failure detected: {}", ownerId, lockFileUri);
+ } else {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ }
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private @Nonnull Pair<LockGetResult, Option<StorageLockFile>> getLockFileFromBlob(BlobClient blobClient, String eTag) {
+ try (InputStream inputStream = blobClient.openInputStream()) {
+ return Pair.of(LockGetResult.SUCCESS, Option.of(StorageLockFile.createFromStream(inputStream, eTag)));
+ } catch (IOException ioe) {
+ throw new UncheckedIOException("Failed reading blob: " + lockFileUri, ioe);
+ }
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getValue() != null ? response.getValue().getETag() : null;
+ if (newEtag == null) {
+ newEtag = lockBlobClient.getProperties().getETag();
+ }
+ return new StorageLockFile(lockData, newEtag);
+ }
+
+ private LockUpsertResult handleUpsertBlobStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == PRECONDITION_FAILURE_ERROR_CODE || e.getErrorCode() == BlobErrorCode.CONDITION_NOT_MET) {
+ logger.info("OwnerId: {}, Unable to write new lock file. Another process has modified this lockfile {} already.",
+ ownerId, lockFileUri);
+ return LockUpsertResult.ACQUIRED_BY_OTHERS;
+ } else if (code == CONFLICT_ERROR_CODE) {
+ logger.info("OwnerId: {}, Retriable conditional request conflict error: {}", ownerId, lockFileUri);
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ logger.warn("OwnerId: {}, Error writing lock file: {}", ownerId, lockFileUri, e);
+ }
+ return LockUpsertResult.UNKNOWN_ERROR;
+ }
+ @Override
+ public Option<String> readObject(String filePath, boolean checkExistsFirst) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ AzureLocation lockLocation = parseAzureLocation(lockFileUri);
+ BlobServiceClient svc = location.blobEndpoint.equals(lockLocation.blobEndpoint)
+ ? blobServiceClient
+ : createDefaultBlobServiceClient().apply(location.withProperties(clientProperties));
+ BlobClient blobClient = svc.getBlobContainerClient(location.container).getBlobClient(location.blobPath);
+
+ if (checkExistsFirst && !blobClient.exists()) {
+ logger.debug("JSON config file not found: {}", filePath);
+ return Option.empty();
+ }
+ byte[] bytes = blobClient.downloadContent().toBytes();
+ return Option.of(new String(bytes, UTF_8));
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() == NOT_FOUND_ERROR_CODE) {
+ logger.debug("JSON config file not found: {}", filePath);
+ } else {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ }
+ return Option.empty();
+ } catch (Exception e) {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ return Option.empty();
+ }
+ }
+
+ @Override
+ public boolean writeObject(String filePath, String content) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ AzureLocation lockLocation = parseAzureLocation(lockFileUri);
+ BlobServiceClient svc = location.blobEndpoint.equals(lockLocation.blobEndpoint)
+ ? blobServiceClient
+ : createDefaultBlobServiceClient().apply(location.withProperties(clientProperties));
+ BlobClient blobClient = svc.getBlobContainerClient(location.container).getBlobClient(location.blobPath);
+ blobClient.upload(BinaryData.fromString(content), true);
+ logger.debug("Successfully wrote object to: {}", filePath);
+ return true;
+ } catch (Exception e) {
+ logger.error("Error writing object to: {}", filePath, e);
+ return false;
+ }
+ }
+
+ @Override
+ public void close() {
+ // BlobServiceClient does not require explicit close. No-op.
+ }
+
+ @VisibleForTesting
+ static AzureLocation parseAzureLocation(String uriString) {
+ try {
+ URI uri = new URI(uriString);
+ String scheme = uri.getScheme();
+ if (scheme == null) {
+ throw new IllegalArgumentException("URI does not contain a valid scheme.");
+ }
+
+ String authority = uri.getAuthority();
+ String path = uri.getPath() == null ? "" : uri.getPath().replaceFirst("/", "");
+
+ // ADLS Gen2: abfs[s]://<container>@<account>.dfs.core.windows.net/<path>
+ if ("abfs".equalsIgnoreCase(scheme) || "abfss".equalsIgnoreCase(scheme)) {
+ if (authority == null || !authority.contains("@")) {
+ throw new IllegalArgumentException("ABFS URI authority must be in the form '<container>@<host>': " + uriString);
+ }
+ String[] parts = authority.split("@", 2);
+ String container = parts[0];
+ String host = parts[1];
+ String endpointHost = dfsHostToBlobHost(host);
+ String endpoint = "https://" + endpointHost;
+ if (container.isEmpty() || path.isEmpty()) {
+ throw new IllegalArgumentException("ABFS URI must contain container and path: " + uriString);
+ }
+ return new AzureLocation(endpoint, container, path, null);
+ }
+
+ // Azure Blob Storage: wasb[s]://<container>@<account>.blob.core.windows.net/<path>
+ if ("wasb".equalsIgnoreCase(scheme) || "wasbs".equalsIgnoreCase(scheme)) {
+ if (authority == null || !authority.contains("@")) {
+ throw new IllegalArgumentException("WASB URI authority must be in the form '<container>@<host>': " + uriString);
+ }
+ String[] parts = authority.split("@", 2);
+ String container = parts[0];
+ String host = parts[1];
+ String endpoint = "https://" + host;
+ if (container.isEmpty() || path.isEmpty()) {
+ throw new IllegalArgumentException("WASB URI must contain container and path: " + uriString);
+ }
+ return new AzureLocation(endpoint, container, path, null);
+ }
+
+ // Direct HTTPS: https://<account>.blob.core.windows.net/<container>/<path>
+ if ("https".equalsIgnoreCase(scheme)) {
+ if (authority == null || authority.isEmpty()) {
+ throw new IllegalArgumentException("HTTPS URI authority missing: " + uriString);
+ }
+ int slash = path.indexOf('/');
+ if (slash <= 0 || slash == path.length() - 1) {
+ throw new IllegalArgumentException("HTTPS URI must be in the form 'https://<host>/<container>/<path>': " + uriString);
+ }
+ String container = path.substring(0, slash);
+ String blobPath = path.substring(slash + 1);
+ return new AzureLocation("https://" + authority, container, blobPath, null);
+ }
Review Comment:
Use `BlobUrlParts.parse` for the parsing? Also including `http`?
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
+ } catch (BlobStorageException e) {
+ // Blob can change/disappear after properties call; treat 404 during stream read as UNKNOWN_ERROR.
+ return Pair.of(handleGetStorageException(e, true), Option.empty());
+ }
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e, false), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e, boolean ignore404) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ if (ignore404) {
+ logger.info("OwnerId: {}, Azure stream read failure detected: {}", ownerId, lockFileUri);
+ } else {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ }
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private @Nonnull Pair<LockGetResult, Option<StorageLockFile>> getLockFileFromBlob(BlobClient blobClient, String eTag) {
+ try (InputStream inputStream = blobClient.openInputStream()) {
+ return Pair.of(LockGetResult.SUCCESS, Option.of(StorageLockFile.createFromStream(inputStream, eTag)));
+ } catch (IOException ioe) {
+ throw new UncheckedIOException("Failed reading blob: " + lockFileUri, ioe);
+ }
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getValue() != null ? response.getValue().getETag() : null;
+ if (newEtag == null) {
+ newEtag = lockBlobClient.getProperties().getETag();
+ }
Review Comment:
Why is refetching Etag needed? This breaks the guarantee as it can be changed by other writer.
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/ADLSStorageLockClient.java:
##########
@@ -0,0 +1,447 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.StorageBasedLockConfig;
+
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobProperties;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+
+import javax.annotation.Nonnull;
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Azure Data Lake Storage (ADLS) implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ul>
+ * <li>{@code hoodie.write.lock.azure.connection.string}</li>
+ * <li>{@code hoodie.write.lock.azure.sas.token}</li>
+ * <li>DefaultAzureCredential</li>
+ * </ul>
+ */
+@Slf4j
+@ThreadSafe
+public class ADLSStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = "hoodie.write.lock.azure.connection.string";
+ public static final String AZURE_SAS_TOKEN = "hoodie.write.lock.azure.sas.token";
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public ADLSStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), log);
+ }
+
+ @VisibleForTesting
+ ADLSStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+
+ AzureLocation location = parseAzureLocation(lockFileUri);
+ this.blobServiceClient = blobServiceClientSupplier.apply(location.withProperties(props));
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ return builder.credential(new DefaultAzureCredentialBuilder().build()).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ long validityTimeoutSecs = getLongProperty(props, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ private static long getLongProperty(Properties props, org.apache.hudi.common.config.ConfigProperty<Long> prop) {
+ if (props == null) {
+ return prop.defaultValue();
+ }
+ Object value = props.get(prop.key());
+ if (value == null) {
+ return prop.defaultValue();
+ }
+ try {
+ return Long.parseLong(String.valueOf(value));
+ } catch (NumberFormatException e) {
+ return prop.defaultValue();
+ }
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ BlobProperties props = lockBlobClient.getProperties();
+ String eTag = props.getETag();
+ try {
+ return getLockFileFromBlob(lockBlobClient, eTag);
+ } catch (BlobStorageException e) {
+ // Blob can change/disappear after properties call; treat 404 during stream read as UNKNOWN_ERROR.
+ return Pair.of(handleGetStorageException(e, true), Option.empty());
+ }
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e, false), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e, boolean ignore404) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ if (ignore404) {
+ logger.info("OwnerId: {}, Azure stream read failure detected: {}", ownerId, lockFileUri);
Review Comment:
As mentioned in the other comment, this particular handling is unnecessary if the lock read happens in one call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4148744494
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 828646839bbe115c06e8a01e982e62ac4fc33292 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505)
* 598acc412f2dcbde52a06add8e314d67f80f3ee0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4158171779
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 637a857b02bfb874dcc5eb5bfec909f94d271326 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4157693681
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 598acc412f2dcbde52a06add8e314d67f80f3ee0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899)
* 637a857b02bfb874dcc5eb5bfec909f94d271326 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3956386864
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/17951?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 45.41%. Comparing base ([`ec04479`](https://app.codecov.io/gh/apache/hudi/commit/ec044790fe71f04710472ad63e7859f1b29e3985?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`df7054b`](https://app.codecov.io/gh/apache/hudi/commit/df7054b59932d78d931e443bc3ecbd1e4b9b83b4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 995 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (ec04479) and HEAD (df7054b). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (ec04479) | HEAD (df7054b) |
>|------|------|------|
>||1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17951 +/- ##
=============================================
- Coverage 61.43% 45.41% -16.02%
+ Complexity 23082 8370 -14712
=============================================
Files 2108 1177 -931
Lines 127636 60239 -67397
Branches 14534 6414 -8120
=============================================
- Hits 78409 27357 -51052
+ Misses 42873 29908 -12965
+ Partials 6354 2974 -3380
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/17951/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/17951/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.41% <100.00%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/17951?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...n/java/org/apache/hudi/storage/StorageSchemes.java](https://app.codecov.io/gh/apache/hudi/pull/17951?src=pr&el=tree&filepath=hudi-io%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fstorage%2FStorageSchemes.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1pby9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaHVkaS9zdG9yYWdlL1N0b3JhZ2VTY2hlbWVzLmphdmE=) | `73.21% <100.00%> (ø)` | |
... and [3243 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/17951/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4131274859
Hi @chrevanthreddy any update on addressing the comments in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add hudi azure based storage lock [hudi]
Posted by "chrevanthreddy (via GitHub)" <gi...@apache.org>.
chrevanthreddy commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3770913750
26/01/20 03:44:01 INFO StorageBasedLockProvider: Owner 8d5f9d59-b590-46bf-b0f1-a6f492e179f6: Lock file path abfss://<container>.dfs.core.windows.net/lakehousetables/hudi_nbcc_test_v4/.hoodie/.locks/table_lock.json, Thread Thread[#1966,SparkConnectExecuteThread_opId=a36945a5-b174-400c-be28-17c37fac9988,5,main], Storage based lock state FAILED_TO_ACQUIRE, Lock already held by a47d30ef-f27f-4554-ab54-2ee872f5a806
26/01/20 03:44:02 INFO ActiveTimelineV2: Created new file for toInstant: abfss://<container>.dfs.core.windows.net/lakehousetables/hudi_nbcc_test_v4/.hoodie/timeline/20260120034324550_20260120034401813.deltacommit
26/01/20 03:44:02 INFO ActiveTimelineV2: Completed [==>20260120034324550__deltacommit__INFLIGHT]
26/01/20 03:44:02 INFO HoodieTableConfig: Loading table properties from abfss://<container>.dfs.core.windows.net/lakehousetables/hudi_nbcc_test_v4/.hoodie/hoodie.properties
26/01/20 03:44:02 INFO HoodieSparkIndexClient: Registering or updating index: column_stats of type: column_stats
26/01/20 03:44:02 INFO HoodieIndexUtils: Registering index column_stats of using column_stats
26/01/20 03:44:02 INFO SessionHolder: Session with userId: uid and sessionId: eb178038-e69c-46fb-8bb8-3e4920be4098 accessed,time 1768880642806 ms.
26/01/20 03:44:02 INFO SessionHolder: Session with userId: uid and sessionId: eb178038-e69c-46fb-8bb8-3e4920be4098 accessed,time 1768880642806 ms.
26/01/20 03:44:02 INFO SessionHolder: Session with userId: uid and sessionId: eb178038-e69c-46fb-8bb8-3e4920be4098 accessed,time 1768880642806 ms.
26/01/20 03:44:03 INFO StorageBasedLockProvider: Owner 8d5f9d59-b590-46bf-b0f1-a6f492e179f6: Lock file path abfss://<container>.dfs.core.windows.net/lakehousetables/hudi_nbcc_test_v4/.hoodie/.locks/table_lock.json, Thread Thread[#1966,SparkConnectExecuteThread_opId=a36945a5-b174-400c-be28-17c37fac9988,5,main], Storage based lock state FAILED_TO_ACQUIRE, Lock already held by a47d30ef-f27f-4554-ab54-2ee872f5a806
26/01/20 03:44:03 INFO FSUtils: Removed directory at abfss://<container>.dfs.core.windows.net/lakehousetables/hudi_nbcc_test_v4/.hoodie/.temp/20260120034324550
26/01/20 03:44:03 INFO HoodieHeartbeatClient: Stopping heartbeat for instant 20260120034324550
26/01/20 03:44:03 INFO HoodieHeartbeatClient: Stopped heartbeat for instant 20260120034324550
26/01/20 03:44:03 INFO HeartbeatUtils: Deleted the heartbeat for instant 20260120034324550
26/01/20 03:44:03 INFO HoodieHeartbeatClient: Deleted heartbeat file for instant 20260120034324550
26/01/20 03:44:03 INFO BaseHoodieWriteClient: Committed 20260120034324550
26/01/20 03:44:03 INFO TransactionManager: State change ending for action instant Option{val=[==>20260120034324550__deltacommit__INFLIGHT]}
26/01/20 03:44:03 INFO StorageBasedLockProvider: Owner a47d30ef-f27f-4554-ab54-2ee872f5a806: Lock file path abfss://<container>.dfs.core.windows.net/lakehousetables/hudi_nbcc_test_v4/.hoodie/.locks/table_lock.json, Thread Thread[#1965,SparkConnectExecuteThread_opId=2f499ea1-de10-4b91-a08f-fdd22ba1a72e,5,main], Storage based lock state RELEASED
26/01/20 03:44:03 INFO LockManager: Released connection created for acquiring lock
26/01/20 03:44:03 INFO TransactionManager: State change ended for action instant Option{val=[==>20260120034324550__deltacommit__INFLIGHT]}
26/01/20 03:44:03 INFO MapPartitionsRDD: Removing RDD 572 from persistence list
26/01/20 03:44:03 INFO BlockManager: Removing RDD 572
26/01/20 03:44:03 INFO MapPartitionsRDD: Removing RDD 591 from persistence list
26/01/20 03:44:03 INFO BlockManager: Removing RDD 591
26/01/20 03:44:03 INFO BaseHoodieWriteClient: Start to archive synchronously.
26/01/20 03:44:03 INFO TransactionManager: State change starting for Optional.empty with latest completed action instant Optional.empty
26/01/20 03:44:03 INFO LockManager: LockProvider org.apache.hudi.client.transaction.lock.StorageBasedLockProvider
26/01/20 03:44:03 INFO StorageBasedLockProvider: Instantiated new storage-based lock provider, owner: 21830948-aa12-497a-ad44-96490bd668df, lockfilePath: abfss://<container>.dfs.core.windows.net/lakehousetables/hudi_nbcc_test_v4/.hoodie/.locks/table_lock.json
26/01/20 03:44:04 INFO StorageBasedLockProvider: Owner 21830948-aa12-497a-ad44-96490bd668df: Lock file path abfss://<container>.dfs.core.windows.net/lakehousetables/hudi_nbcc_test_v4/.hoodie/.locks/table_lock.json, Thread Thread[#1965,SparkConnectExecuteThread_opId=2f499ea1-de10-4b91-a08f-fdd22ba1a72e,5,main], Storage based lock state ACQUIRED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4272703430
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
}, {
"hash" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347",
"triggerID" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"triggerType" : "PUSH"
}, {
"hash" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350",
"triggerID" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"triggerType" : "PUSH"
}, {
"hash" : "d2d7cb2002a639a6aeedac2f31c35904fc45b2a8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13353",
"triggerID" : "d2d7cb2002a639a6aeedac2f31c35904fc45b2a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2d7cb2002a639a6aeedac2f31c35904fc45b2a8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13353)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4271842309
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
}, {
"hash" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347",
"triggerID" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"triggerType" : "PUSH"
}, {
"hash" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266)
* 2992185d3b874b02c9c4ec77e6f96c8ee83770d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347)
* 4107d19f79f42fa5e9c587e0e78936b7d8ca0ace UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3773431913
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215",
"triggerID" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"triggerType" : "PUSH"
}, {
"hash" : "318978795585cc0c8a52b5b367cb7fffd81ee949",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "318978795585cc0c8a52b5b367cb7fffd81ee949",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38e4d44e679d3c93a99bb5300834dc54d1b56c3e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215)
* 318978795585cc0c8a52b5b367cb7fffd81ee949 UNKNOWN
* 43b35a0a65113c81463a237015cb99a18515d352 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3769603612
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215",
"triggerID" : "38e4d44e679d3c93a99bb5300834dc54d1b56c3e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 38e4d44e679d3c93a99bb5300834dc54d1b56c3e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11215)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "chrevanthreddy (via GitHub)" <gi...@apache.org>.
chrevanthreddy commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r2854673635
##########
hudi-azure/src/main/java/org/apache/hudi/config/AzureStorageLockConfig.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.common.config.ConfigClassProperty;
+import org.apache.hudi.common.config.ConfigGroups;
+import org.apache.hudi.common.config.ConfigProperty;
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.config.LockConfiguration;
+
+/**
+ * Hoodie Configs for Azure based storage locks.
+ */
+@ConfigClassProperty(name = "Azure based Locks Configurations",
+ groupName = ConfigGroups.Names.WRITE_CLIENT,
+ subGroupName = ConfigGroups.SubGroupNames.LOCK,
+ description = "Configs that control Azure Blob/ADLS based locking mechanisms "
+ + "required for concurrency control between writers to a Hudi table.")
+public class AzureStorageLockConfig extends HoodieConfig {
+
+ private static final String AZURE_BASED_LOCK_PROPERTY_PREFIX = LockConfiguration.LOCK_PREFIX + "azure.";
+
+ public static final ConfigProperty<String> AZURE_CONNECTION_STRING = ConfigProperty
Review Comment:
Updated config docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3874753855
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43b35a0a65113c81463a237015cb99a18515d352 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-3885489641
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 43b35a0a65113c81463a237015cb99a18515d352 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857)
* 24d84063a6e9887312aedb0d1372d28c243a20f5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4272059864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
}, {
"hash" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347",
"triggerID" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"triggerType" : "PUSH"
}, {
"hash" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350",
"triggerID" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266)
* 2992185d3b874b02c9c4ec77e6f96c8ee83770d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347)
* 4107d19f79f42fa5e9c587e0e78936b7d8ca0ace Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r3103351264
##########
hudi-azure/src/main/java/org/apache/hudi/azure/credentials/AzureCredentialFactory.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.credentials;
+
+import org.apache.hudi.config.AzureStorageLockConfig;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.ClientSecretCredentialBuilder;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.identity.ManagedIdentityCredentialBuilder;
+
+import java.util.Properties;
+
+/**
+ * Factory for resolving an Azure {@link TokenCredential} from Hudi properties.
+ *
+ * <p>Credential precedence:
+ * <ol>
+ * <li>User-assigned managed identity ({@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID})
+ * — uses {@code ManagedIdentityCredential}</li>
+ * <li>Service principal ({@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET})
+ * — uses {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — (system-assigned MI,
+ * workload identity, env-var SP, Azure CLI, etc.); suitable for dev and environments
+ * where auth is controlled externally</li>
+ * </ol>
+ *
+ * <p>Note: connection string and SAS token auth are not {@link TokenCredential}-based and are
+ * handled directly by the caller before this factory is consulted.
+ */
+public class AzureCredentialFactory {
+
+ /**
+ * Lazily initializes {@code DefaultAzureCredential} on first use of the default chain only.
+ */
+ private static final class DefaultAzureCredentialHolder {
+ static final TokenCredential INSTANCE = new DefaultAzureCredentialBuilder().build();
+ }
+
+ private AzureCredentialFactory() {
+ }
+
+ /**
+ * Returns a {@link TokenCredential} resolved from the supplied properties.
+ *
+ * @param props Hudi lock properties
+ * @return resolved credential, never {@code null}
+ */
+ public static TokenCredential getAzureCredential(Properties props) {
+ if (props != null) {
+ String miClientId = props.getProperty(AzureStorageLockConfig.AZURE_MANAGED_IDENTITY_CLIENT_ID.key());
Review Comment:
Non-blocker: use `ConfigUtils#getRawValueWithAltKeys` to allow alternative config keys without invoking `.key()`
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+
+ // 2. SAS token.
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ // 3. TokenCredential — MI, service principal, or DefaultAzureCredential fallback.
+ return builder.credential(AzureCredentialFactory.getAzureCredential(props)).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ String eTag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ StorageLockFile lockFile = StorageLockFile.createFromStream(response.getValue().toStream(), eTag);
+ return Pair.of(LockGetResult.SUCCESS, Option.of(lockFile));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ if (newEtag == null && response.getValue() != null) {
+ newEtag = response.getValue().getETag();
Review Comment:
This is handled in a consistent way now.
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_CONNECTION_STRING, true);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+
+ // 2. SAS token.
+ String sasToken = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_SAS_TOKEN, true);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ // 3. TokenCredential — MI, service principal, or DefaultAzureCredential fallback.
+ return builder.credential(AzureCredentialFactory.getAzureCredential(props)).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
Review Comment:
nit: would be good to log the invalid format of the config so user knows.
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_CONNECTION_STRING, true);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+
+ // 2. SAS token.
+ String sasToken = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_SAS_TOKEN, true);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ // 3. TokenCredential — MI, service principal, or DefaultAzureCredential fallback.
+ return builder.credential(AzureCredentialFactory.getAzureCredential(props)).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent()
+ ? previousLockFile.get().getVersionId()
+ : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ //Check for null or empty ETag and inconsistent quotes
+ String eTag = canonicalizeEtag(
+ response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null,
+ "download");
+ StorageLockFile lockFile = StorageLockFile.createFromStream(response.getValue().toStream(), eTag);
+ return Pair.of(LockGetResult.SUCCESS, Option.of(lockFile));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ if (newEtag == null && response.getValue() != null) {
+ newEtag = response.getValue().getETag();
+ }
+ //Check for null or empty ETag and inconsistent quotes
+ newEtag = canonicalizeEtag(newEtag, "upload");
+ return new StorageLockFile(lockData, newEtag);
+ }
+
+ private String canonicalizeEtag(String eTag, String operation) {
+ if (eTag == null) {
+ throw new HoodieLockException("Missing ETag in Azure " + operation + " response for lock file: " + lockFileUri);
+ }
+
+ String normalized = eTag.trim();
+ if (normalized.isEmpty()) {
+ throw new HoodieLockException("Missing ETag in Azure " + operation + " response for lock file: " + lockFileUri);
+ }
+
+ boolean startsWithQuote = normalized.startsWith("\"");
+ boolean endsWithQuote = normalized.endsWith("\"");
+ if (startsWithQuote && endsWithQuote) {
+ return normalized;
+ }
+ if (!startsWithQuote && !endsWithQuote) {
+ return "\"" + normalized + "\"";
+ }
+
+ throw new HoodieLockException("Malformed ETag in Azure " + operation + " response for lock file: " + lockFileUri);
+ }
+
+ private LockUpsertResult handleUpsertBlobStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == PRECONDITION_FAILURE_ERROR_CODE || e.getErrorCode() == BlobErrorCode.CONDITION_NOT_MET) {
+ logger.info("OwnerId: {}, Unable to write new lock file. Another process has modified this lockfile {} already.",
+ ownerId, lockFileUri);
+ return LockUpsertResult.ACQUIRED_BY_OTHERS;
+ } else if (code == CONFLICT_ERROR_CODE) {
+ logger.info("OwnerId: {}, Retriable conditional request conflict error: {}", ownerId, lockFileUri);
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ logger.warn("OwnerId: {}, Error writing lock file: {}", ownerId, lockFileUri, e);
+ }
+ return LockUpsertResult.UNKNOWN_ERROR;
+ }
+
+ @Override
+ public Option<String> readObject(String filePath, boolean checkExistsFirst) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ BlobServiceClient svc = getBlobServiceClient(location);
+ BlobClient blobClient = svc.getBlobContainerClient(location.container).getBlobClient(location.blobPath);
+
+ if (checkExistsFirst && !blobClient.exists()) {
+ logger.debug("JSON config file not found: {}", filePath);
+ return Option.empty();
+ }
+ byte[] bytes = blobClient.downloadContent().toBytes();
+ return Option.of(new String(bytes, UTF_8));
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() == NOT_FOUND_ERROR_CODE) {
+ logger.debug("JSON config file not found: {}", filePath);
+ } else {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ }
+ return Option.empty();
+ } catch (Exception e) {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ return Option.empty();
+ }
+ }
+
+ @Override
+ public boolean writeObject(String filePath, String content) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ BlobServiceClient svc = getBlobServiceClient(location);
+ BlobClient blobClient = svc.getBlobContainerClient(location.container).getBlobClient(location.blobPath);
+ blobClient.upload(BinaryData.fromString(content), true);
+ logger.debug("Successfully wrote object to: {}", filePath);
+ return true;
+ } catch (Exception e) {
+ logger.error("Error writing object to: {}", filePath, e);
+ return false;
+ }
+ }
+
+ private BlobServiceClient getBlobServiceClient(AzureLocation location) {
+ if (location.blobEndpoint.equals(lockBlobEndpoint)) {
+ return blobServiceClient;
+ }
+ return secondaryBlobServiceClients.computeIfAbsent(
+ location.blobEndpoint,
+ endpoint -> blobServiceClientSupplier.apply(location.withProperties(clientProperties)));
+ }
Review Comment:
Not an issue
##########
hudi-azure/src/main/java/org/apache/hudi/azure/credentials/AzureCredentialFactory.java:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.credentials;
+
+import org.apache.hudi.config.AzureStorageLockConfig;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.ClientSecretCredentialBuilder;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.identity.ManagedIdentityCredentialBuilder;
+
+import java.util.Properties;
+
+/**
+ * Factory for resolving an Azure {@link TokenCredential} from Hudi properties.
+ *
+ * <p>Credential precedence:
+ * <ol>
+ * <li>User-assigned managed identity ({@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID})
+ * — uses {@code ManagedIdentityCredential}</li>
+ * <li>Service principal ({@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET})
+ * — uses {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — (system-assigned MI,
+ * workload identity, env-var SP, Azure CLI, etc.); suitable for dev and environments
+ * where auth is controlled externally</li>
+ * </ol>
+ *
+ * <p>Note: connection string and SAS token auth are not {@link TokenCredential}-based and are
+ * handled directly by the caller before this factory is consulted.
+ */
+public class AzureCredentialFactory {
+
+ // Shared instance so the token cache and IMDS probe are reused across all clients
+ // that fall through to the default chain.
+ private static final TokenCredential DEFAULT_AZURE_CREDENTIAL =
+ new DefaultAzureCredentialBuilder().build();
+
Review Comment:
This is fixed now.
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_CONNECTION_STRING, true);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+
+ // 2. SAS token.
+ String sasToken = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_SAS_TOKEN, true);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ // 3. TokenCredential — MI, service principal, or DefaultAzureCredential fallback.
+ return builder.credential(AzureCredentialFactory.getAzureCredential(props)).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent()
+ ? previousLockFile.get().getVersionId()
+ : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ //Check for null or empty ETag and inconsistent quotes
+ String eTag = canonicalizeEtag(
+ response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null,
+ "download");
+ StorageLockFile lockFile = StorageLockFile.createFromStream(response.getValue().toStream(), eTag);
+ return Pair.of(LockGetResult.SUCCESS, Option.of(lockFile));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e), Option.empty());
+ }
+ }
Review Comment:
This is a minor not required to be addressed.
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_CONNECTION_STRING = AzureStorageLockConfig.AZURE_CONNECTION_STRING.key();
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = props == null ? null : props.getProperty(AZURE_CONNECTION_STRING);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+
+ // 2. SAS token.
+ String sasToken = props == null ? null : props.getProperty(AZURE_SAS_TOKEN);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ // 3. TokenCredential — MI, service principal, or DefaultAzureCredential fallback.
+ return builder.credential(AzureCredentialFactory.getAzureCredential(props)).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent() ? previousLockFile.get().getVersionId() : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
Review Comment:
This is fixed now.
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
Review Comment:
This is legit nit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4252874491
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 637a857b02bfb874dcc5eb5bfec909f94d271326 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932)
* 2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #17951:
URL: https://github.com/apache/hudi/pull/17951#discussion_r3087264370
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
Review Comment:
<a href="#"><img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=7" align="top"></a> **Redundant public constant pollutes API**
`AZURE_SAS_TOKEN` duplicates `AzureStorageLockConfig.AZURE_SAS_TOKEN.key()` and is never used internally — the class references `AzureStorageLockConfig.AZURE_SAS_TOKEN` directly at line 177. Exposing it as a `public static final` string in this class creates a secondary API surface that callers might depend on, making future refactoring harder without a clear benefit.
Remove the constant entirely; anyone needing the key should use `AzureStorageLockConfig.AZURE_SAS_TOKEN.key()` directly.
— *Greptile* ([original](https://github.com/yihua/hudi/pull/45#discussion_r3087235472)) (source:comment#3087235472)
##########
hudi-azure/pom.xml:
##########
@@ -0,0 +1,240 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <parent>
+ <artifactId>hudi</artifactId>
+ <groupId>org.apache.hudi</groupId>
+ <version>1.2.0-SNAPSHOT</version>
+ <relativePath>../pom.xml</relativePath>
+ </parent>
+
+ <modelVersion>4.0.0</modelVersion>
+ <artifactId>hudi-azure</artifactId>
+ <packaging>jar</packaging>
+
+ <properties>
+ <!-- Azure SDK versions are intentionally pinned here to keep this module self-contained. -->
+ <azure.storage.blob.version>12.26.0</azure.storage.blob.version>
+ <azure.identity.version>1.12.2</azure.identity.version>
+ <!-- Keep Jackson consistent for Azure SDK integration tests (Hudi core pins older Jackson versions). -->
+ <azure.jackson.version>2.13.5</azure.jackson.version>
Review Comment:
_⚠️ Potential issue_ | _🟠 Major_
<details>
<summary>🧩 Analysis chain</summary>
🏁 Script executed:
```shell
#!/bin/bash
set -euo pipefail
# Verify vulnerabilities for jackson-core:2.13.5 from OSV
curl -s https://api.osv.dev/v1/query \
-H "Content-Type: application/json" \
-d '{"package":{"ecosystem":"Maven","name":"com.fasterxml.jackson.core:jackson-core"},"version":"2.13.5"}' \
| jq '{vuln_ids: [.vulns[].id], summaries: [.vulns[].summary]}'
```
Repository: yihua/hudi
Length of output: 339
---
🏁 Script executed:
```shell
cat -n hudi-azure/pom.xml | head -150
```
Repository: yihua/hudi
Length of output: 6482
---
🌐 Web query:
`Jackson 2.13 latest patched version CVE GHSA-72hv-8253-57qq`
💡 Result:
The official FasterXML Jackson 2.13 branch latest version is 2.13.5 (released around 2023), with no further patches. The GHSA-72hv-8253-57qq vulnerability (Number Length Constraint Bypass in Async Parser, published Feb 26, 2026, affects jackson-core 2.0.0-2.18.5 and others, patched officially in 2.18.6+) is not patched in official 2.13.x releases, as the branch is closed. Third-party patches exist: - HeroDevs NES: jackson-core 2.13.5-jackson-core-2.13.7 (Mar 3, 2026), explicitly patches GHSA-72hv-8253-57qq, available via their Maven registry (https://registry.nes.herodevs.com/maven). - sassoftware/jackson: Custom branch 2.13.5-CVE-2025-52999-sonatype-2022-6438 remediates related issues but not confirmed for this specific GHSA. For official sources, upgrade to Jackson 2.18.6+ is recommended. HeroDevs provides the latest patched 2.13 variant if stuck on 2.13.
Citations:
- 1: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.13
- 2: https://docs.herodevs.com/jackson/release-notes/jackson-core-2-13
- 3: https://github.com/FasterXML/jackson-core/security/advisories/GHSA-72hv-8253-57qq
- 4: https://docs.herodevs.com/jackson/release-notes/jackson-core
- 5: https://docs.herodevs.com/jackson/additional-info/maven-packages
- 6: http://docs.herodevs.com/jackson/2-13/jackson
---
</details>
**Upgrade Jackson from 2.13.5 to address known vulnerabilities (GHSA-72hv-8253-57qq, GHSA-h46c-h94j-95f3).**
`azure.jackson.version` is pinned to 2.13.5, which contains two high-severity DoS vulnerabilities. The Jackson 2.13.x branch is no longer maintained; upgrade to **2.18.6 or later** if Azure SDK compatibility permits. If locked on 2.13.x, use HeroDevs NES patched version 2.13.5-jackson-core-2.13.7 (published Mar 3, 2026, which explicitly patches GHSA-72hv-8253-57qq).
Applies to all Jackson artifacts in lines 118–147 that inherit this version.
<details>
<summary>🤖 Prompt for AI Agents</summary>
```
Verify each finding against the current code and only fix it if needed.
In `@hudi-azure/pom.xml` at line 37, The pom's azure.jackson.version property is
pinned to vulnerable 2.13.5; update the property azure.jackson.version to a
secure release (preferably 2.18.6 or later) so all Jackson artifacts that
inherit this property (the Jackson dependencies declared under the pom) are
upgraded; if you cannot move off 2.13.x for Azure SDK compatibility, set
azure.jackson.version to the NES patched coordinate 2.13.5-jackson-core-2.13.7
instead; ensure you run a full build to confirm compatibility after changing the
azure.jackson.version property.
```
</details>
<!-- fingerprinting:phantom:poseidon:hawk -->
<!-- This is an auto-generated comment by CodeRabbit -->
— *CodeRabbit* ([original](https://github.com/yihua/hudi/pull/45#discussion_r3087262692)) (source:comment#3087262692)
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_CONNECTION_STRING, true);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+
+ // 2. SAS token.
+ String sasToken = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_SAS_TOKEN, true);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ // 3. TokenCredential — MI, service principal, or DefaultAzureCredential fallback.
+ return builder.credential(AzureCredentialFactory.getAzureCredential(props)).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent()
+ ? previousLockFile.get().getVersionId()
+ : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ //Check for null or empty ETag and inconsistent quotes
+ String eTag = canonicalizeEtag(
+ response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null,
+ "download");
+ StorageLockFile lockFile = StorageLockFile.createFromStream(response.getValue().toStream(), eTag);
+ return Pair.of(LockGetResult.SUCCESS, Option.of(lockFile));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e), Option.empty());
+ }
+ }
Review Comment:
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=7" align="top"></a> **`readCurrentLockFile` can throw `HoodieLockException` past an unguarded caller**
`canonicalizeEtag` (called at line 244) throws `HoodieLockException` when the ETag is null, empty, or malformed. Only `BlobStorageException` is caught here, so that unchecked exception escapes the method. `StorageBasedLockProvider.readCurrentLockFile` (line 326 in `StorageBasedLockProvider.java`) calls this with no surrounding try-catch, meaning a corrupt Azure HTTP response will propagate as an unhandled exception and surface as a lock-acquisition failure rather than being gracefully retried.
By contrast, `tryUpsertLockFile` catches every `Exception` and maps it to `LockUpsertResult.UNKNOWN_ERROR`. The same pattern applied here would keep behaviour consistent with the interface's return-value contract:
```suggestion
@Override
public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
try {
Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
//Check for null or empty ETag and inconsistent quotes
String eTag = canonicalizeEtag(
response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null,
"download");
StorageLockFile lockFile = StorageLockFile.createFromStream(response.getValue().toStream(), eTag);
return Pair.of(LockGetResult.SUCCESS, Option.of(lockFile));
} catch (BlobStorageException e) {
return Pair.of(handleGetStorageException(e), Option.empty());
} catch (HoodieLockException e) {
logger.error("OwnerId: {}, Malformed or missing ETag in Azure response for lock file: {}", ownerId, lockFileUri, e);
throw e;
} catch (Exception e) {
logger.error("OwnerId: {}, Unexpected error while reading lock file: {}", ownerId, lockFileUri, e);
return Pair.of(LockGetResult.UNKNOWN_ERROR, Option.empty());
}
}
```
At minimum, the explicit re-throw of `HoodieLockException` (with logging) makes the failure mode obvious at the call site and clearly documents the intentional "fail fast on protocol violation" behaviour.
— *Greptile* ([original](https://github.com/yihua/hudi/pull/45#discussion_r3087235260)) (source:comment#3087235260)
##########
hudi-azure/src/main/java/org/apache/hudi/azure/transaction/lock/AzureStorageLockClient.java:
##########
@@ -0,0 +1,469 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.azure.transaction.lock;
+
+import org.apache.hudi.client.transaction.lock.StorageLockClient;
+import org.apache.hudi.client.transaction.lock.models.LockGetResult;
+import org.apache.hudi.client.transaction.lock.models.LockUpsertResult;
+import org.apache.hudi.client.transaction.lock.models.StorageLockData;
+import org.apache.hudi.client.transaction.lock.models.StorageLockFile;
+import org.apache.hudi.common.util.Functions;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.VisibleForTesting;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieLockException;
+import org.apache.hudi.config.AzureStorageLockConfig;
+import org.apache.hudi.config.StorageBasedLockConfig;
+import org.apache.hudi.common.config.TypedProperties;
+
+import org.apache.hudi.azure.credentials.AzureCredentialFactory;
+
+import com.azure.core.credential.AzureSasCredential;
+import com.azure.core.exception.HttpResponseException;
+import com.azure.core.http.policy.ExponentialBackoffOptions;
+import com.azure.core.http.policy.RetryOptions;
+import com.azure.core.http.rest.Response;
+import com.azure.core.util.BinaryData;
+import com.azure.core.util.Context;
+import com.azure.core.util.HttpClientOptions;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.BlobUrlParts;
+import com.azure.storage.blob.models.BlobErrorCode;
+import com.azure.storage.blob.models.BlobRequestConditions;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.BlockBlobItem;
+import com.azure.storage.blob.options.BlobParallelUploadOptions;
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.Properties;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.hudi.common.util.ConfigUtils.getLongWithAltKeys;
+import static org.apache.hudi.common.util.ConfigUtils.getStringWithAltKeys;
+
+/**
+ * Azure Storage implementation of {@link StorageLockClient} using Azure Blob conditional requests.
+ *
+ * <p>Supports the following URI schemes:
+ * <ul>
+ * <li>ADLS Gen2: {@code abfs://} and {@code abfss://}</li>
+ * <li>Azure Blob Storage: {@code wasb://} and {@code wasbs://}</li>
+ * </ul>
+ *
+ *
+ * <ul>
+ * <li>Create: conditional write with If-None-Match: *</li>
+ * <li>Update/Renew/Expire: conditional write with If-Match: <etag></li>
+ * </ul>
+ *
+ * <p>Expected lock URI formats:
+ * <ul>
+ * <li>{@code abfs://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code abfss://<container>@<account>.dfs.core.windows.net/<path>}</li>
+ * <li>{@code wasb://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * <li>{@code wasbs://<container>@<account>.blob.core.windows.net/<path>}</li>
+ * </ul>
+ *
+ * <p>Authentication precedence (via {@link Properties}):
+ * <ol>
+ * <li>{@link AzureStorageLockConfig#AZURE_CONNECTION_STRING} — connection string (includes shared key)</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_SAS_TOKEN} — shared access signature</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_MANAGED_IDENTITY_CLIENT_ID} — user-assigned managed identity
+ * via {@code ManagedIdentityCredential}</li>
+ * <li>{@link AzureStorageLockConfig#AZURE_CLIENT_TENANT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_ID} +
+ * {@link AzureStorageLockConfig#AZURE_CLIENT_SECRET} — service principal
+ * via {@code ClientSecretCredential}</li>
+ * <li>{@code DefaultAzureCredential} — probing chain; see {@link org.apache.hudi.azure.credentials.AzureCredentialFactory}</li>
+ * </ol>
+ */
+@ThreadSafe
+public class AzureStorageLockClient implements StorageLockClient {
+
+ private static final int PRECONDITION_FAILURE_ERROR_CODE = 412;
+ private static final int NOT_FOUND_ERROR_CODE = 404;
+ private static final int CONFLICT_ERROR_CODE = 409;
+ private static final int RATE_LIMIT_ERROR_CODE = 429;
+ private static final int INTERNAL_SERVER_ERROR_CODE_MIN = 500;
+
+ public static final String AZURE_SAS_TOKEN = AzureStorageLockConfig.AZURE_SAS_TOKEN.key();
+
+ private final Logger logger;
+ private final BlobServiceClient blobServiceClient;
+ private final Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier;
+ private final ConcurrentMap<String, BlobServiceClient> secondaryBlobServiceClients;
+ private final BlobClient lockBlobClient;
+ private final Properties clientProperties;
+ private final String ownerId;
+ private final String lockFileUri;
+ private final String lockBlobEndpoint;
+
+ /**
+ * Constructor used by reflection by {@link org.apache.hudi.client.transaction.lock.StorageBasedLockProvider}.
+ *
+ * @param ownerId lock owner id
+ * @param lockFileUri lock file URI (abfs/abfss/wasb/wasbs)
+ * @param props properties used to customize/authenticate the Azure client
+ */
+ public AzureStorageLockClient(String ownerId, String lockFileUri, Properties props) {
+ this(ownerId, lockFileUri, props, createDefaultBlobServiceClient(), LoggerFactory.getLogger(AzureStorageLockClient.class));
+ }
+
+ @VisibleForTesting
+ AzureStorageLockClient(
+ String ownerId,
+ String lockFileUri,
+ Properties props,
+ Functions.Function1<AzureLocation, BlobServiceClient> blobServiceClientSupplier,
+ Logger logger) {
+ this.ownerId = ownerId;
+ this.lockFileUri = lockFileUri;
+ this.logger = logger;
+ this.clientProperties = props;
+ this.blobServiceClientSupplier = blobServiceClientSupplier;
+ this.secondaryBlobServiceClients = new ConcurrentHashMap<>();
+
+ AzureLocation location = parseAzureLocation(lockFileUri).withProperties(props);
+ this.lockBlobEndpoint = location.blobEndpoint;
+ this.blobServiceClient = blobServiceClientSupplier.apply(location);
+ BlobContainerClient containerClient = blobServiceClient.getBlobContainerClient(location.container);
+ this.lockBlobClient = containerClient.getBlobClient(location.blobPath);
+ }
+
+ private static Functions.Function1<AzureLocation, BlobServiceClient> createDefaultBlobServiceClient() {
+ return (location) -> {
+ Properties props = location.props;
+ BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+ configureAzureClientOptions(builder, props);
+
+ // 1. Connection string (includes shared-key auth).
+ String connectionString = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_CONNECTION_STRING, true);
+ if (connectionString != null && !connectionString.trim().isEmpty()) {
+ return builder.connectionString(connectionString).buildClient();
+ }
+
+ builder.endpoint(location.blobEndpoint);
+
+ // 2. SAS token.
+ String sasToken = getStringWithAltKeys(props, AzureStorageLockConfig.AZURE_SAS_TOKEN, true);
+ if (sasToken != null && !sasToken.trim().isEmpty()) {
+ String cleaned = sasToken.startsWith("?") ? sasToken.substring(1) : sasToken;
+ return builder.credential(new AzureSasCredential(cleaned)).buildClient();
+ }
+
+ // 3. TokenCredential — MI, service principal, or DefaultAzureCredential fallback.
+ return builder.credential(AzureCredentialFactory.getAzureCredential(props)).buildClient();
+ };
+ }
+
+ private static void configureAzureClientOptions(BlobServiceClientBuilder builder, Properties props) {
+ // Set Azure SDK timeouts based on lock validity to avoid long-hanging calls.
+ TypedProperties typedProps = new TypedProperties();
+ if (props != null) {
+ typedProps.putAll(props);
+ }
+ long validityTimeoutSecs;
+ try {
+ validityTimeoutSecs = getLongWithAltKeys(typedProps, StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS);
+ } catch (NumberFormatException e) {
+ validityTimeoutSecs = StorageBasedLockConfig.VALIDITY_TIMEOUT_SECONDS.defaultValue();
+ }
+ long azureCallTimeoutSecs = Math.max(1, validityTimeoutSecs / 5);
+
+ // Disable automatic SDK retries; Hudi manages retries at the lock-provider level.
+ ExponentialBackoffOptions exponentialOptions = new ExponentialBackoffOptions().setMaxRetries(0);
+ RetryOptions retryOptions = new RetryOptions(exponentialOptions);
+
+ HttpClientOptions clientOptions = new HttpClientOptions()
+ .setResponseTimeout(Duration.ofSeconds(azureCallTimeoutSecs))
+ .setReadTimeout(Duration.ofSeconds(azureCallTimeoutSecs));
+
+ builder.retryOptions(retryOptions).clientOptions(clientOptions);
+ }
+
+ @Override
+ public Pair<LockUpsertResult, Option<StorageLockFile>> tryUpsertLockFile(
+ StorageLockData newLockData,
+ Option<StorageLockFile> previousLockFile) {
+ String expectedEtag = previousLockFile.isPresent()
+ ? previousLockFile.get().getVersionId()
+ : null;
+ try {
+ StorageLockFile updated = createOrUpdateLockFileInternal(newLockData, expectedEtag);
+ return Pair.of(LockUpsertResult.SUCCESS, Option.of(updated));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleUpsertBlobStorageException(e), Option.empty());
+ } catch (HttpResponseException e) {
+ logger.error("OwnerId: {}, Unexpected Azure SDK error while writing lock file: {}",
+ ownerId, lockFileUri, e);
+ if (!previousLockFile.isPresent()) {
+ // For create, fail fast since this indicates a larger issue.
+ throw e;
+ }
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ } catch (Exception e) {
+ logger.error("OwnerId: {}, Unexpected error while writing lock file: {}", ownerId, lockFileUri, e);
+ return Pair.of(LockUpsertResult.UNKNOWN_ERROR, Option.empty());
+ }
+ }
+
+ @Override
+ public Pair<LockGetResult, Option<StorageLockFile>> readCurrentLockFile() {
+ try {
+ Response<BinaryData> response = lockBlobClient.downloadContentWithResponse(null, null, null, Context.NONE);
+ //Check for null or empty ETag and inconsistent quotes
+ String eTag = canonicalizeEtag(
+ response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null,
+ "download");
+ StorageLockFile lockFile = StorageLockFile.createFromStream(response.getValue().toStream(), eTag);
+ return Pair.of(LockGetResult.SUCCESS, Option.of(lockFile));
+ } catch (BlobStorageException e) {
+ return Pair.of(handleGetStorageException(e), Option.empty());
+ }
+ }
+
+ private LockGetResult handleGetStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == NOT_FOUND_ERROR_CODE || e.getErrorCode() == BlobErrorCode.BLOB_NOT_FOUND) {
+ logger.info("OwnerId: {}, Object not found in the path: {}", ownerId, lockFileUri);
+ return LockGetResult.NOT_EXISTS;
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ throw e;
+ }
+ return LockGetResult.UNKNOWN_ERROR;
+ }
+
+ private StorageLockFile createOrUpdateLockFileInternal(StorageLockData lockData, String expectedEtag) {
+ byte[] bytes = StorageLockFile.toByteArray(lockData);
+ BlobRequestConditions conditions = new BlobRequestConditions();
+ if (expectedEtag == null) {
+ conditions.setIfNoneMatch("*");
+ } else {
+ conditions.setIfMatch(expectedEtag);
+ }
+
+ BlobParallelUploadOptions options = new BlobParallelUploadOptions(BinaryData.fromBytes(bytes))
+ .setRequestConditions(conditions);
+ Response<BlockBlobItem> response = lockBlobClient.uploadWithResponse(options, null, Context.NONE);
+ String newEtag = response.getHeaders() != null ? response.getHeaders().getValue("ETag") : null;
+ if (newEtag == null && response.getValue() != null) {
+ newEtag = response.getValue().getETag();
+ }
+ //Check for null or empty ETag and inconsistent quotes
+ newEtag = canonicalizeEtag(newEtag, "upload");
+ return new StorageLockFile(lockData, newEtag);
+ }
+
+ private String canonicalizeEtag(String eTag, String operation) {
+ if (eTag == null) {
+ throw new HoodieLockException("Missing ETag in Azure " + operation + " response for lock file: " + lockFileUri);
+ }
+
+ String normalized = eTag.trim();
+ if (normalized.isEmpty()) {
+ throw new HoodieLockException("Missing ETag in Azure " + operation + " response for lock file: " + lockFileUri);
+ }
+
+ boolean startsWithQuote = normalized.startsWith("\"");
+ boolean endsWithQuote = normalized.endsWith("\"");
+ if (startsWithQuote && endsWithQuote) {
+ return normalized;
+ }
+ if (!startsWithQuote && !endsWithQuote) {
+ return "\"" + normalized + "\"";
+ }
+
+ throw new HoodieLockException("Malformed ETag in Azure " + operation + " response for lock file: " + lockFileUri);
+ }
+
+ private LockUpsertResult handleUpsertBlobStorageException(BlobStorageException e) {
+ int code = e.getStatusCode();
+ if (code == PRECONDITION_FAILURE_ERROR_CODE || e.getErrorCode() == BlobErrorCode.CONDITION_NOT_MET) {
+ logger.info("OwnerId: {}, Unable to write new lock file. Another process has modified this lockfile {} already.",
+ ownerId, lockFileUri);
+ return LockUpsertResult.ACQUIRED_BY_OTHERS;
+ } else if (code == CONFLICT_ERROR_CODE) {
+ logger.info("OwnerId: {}, Retriable conditional request conflict error: {}", ownerId, lockFileUri);
+ } else if (code == RATE_LIMIT_ERROR_CODE) {
+ logger.warn("OwnerId: {}, Rate limit exceeded for lock file: {}", ownerId, lockFileUri);
+ } else if (code >= INTERNAL_SERVER_ERROR_CODE_MIN) {
+ logger.warn("OwnerId: {}, Azure returned internal server error code for lock file: {}", ownerId, lockFileUri, e);
+ } else {
+ logger.warn("OwnerId: {}, Error writing lock file: {}", ownerId, lockFileUri, e);
+ }
+ return LockUpsertResult.UNKNOWN_ERROR;
+ }
+
+ @Override
+ public Option<String> readObject(String filePath, boolean checkExistsFirst) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ BlobServiceClient svc = getBlobServiceClient(location);
+ BlobClient blobClient = svc.getBlobContainerClient(location.container).getBlobClient(location.blobPath);
+
+ if (checkExistsFirst && !blobClient.exists()) {
+ logger.debug("JSON config file not found: {}", filePath);
+ return Option.empty();
+ }
+ byte[] bytes = blobClient.downloadContent().toBytes();
+ return Option.of(new String(bytes, UTF_8));
+ } catch (BlobStorageException e) {
+ if (e.getStatusCode() == NOT_FOUND_ERROR_CODE) {
+ logger.debug("JSON config file not found: {}", filePath);
+ } else {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ }
+ return Option.empty();
+ } catch (Exception e) {
+ logger.warn("Error reading JSON config file: {}", filePath, e);
+ return Option.empty();
+ }
+ }
+
+ @Override
+ public boolean writeObject(String filePath, String content) {
+ try {
+ AzureLocation location = parseAzureLocation(filePath);
+ BlobServiceClient svc = getBlobServiceClient(location);
+ BlobClient blobClient = svc.getBlobContainerClient(location.container).getBlobClient(location.blobPath);
+ blobClient.upload(BinaryData.fromString(content), true);
+ logger.debug("Successfully wrote object to: {}", filePath);
+ return true;
+ } catch (Exception e) {
+ logger.error("Error writing object to: {}", filePath, e);
+ return false;
+ }
+ }
+
+ private BlobServiceClient getBlobServiceClient(AzureLocation location) {
+ if (location.blobEndpoint.equals(lockBlobEndpoint)) {
+ return blobServiceClient;
+ }
+ return secondaryBlobServiceClients.computeIfAbsent(
+ location.blobEndpoint,
+ endpoint -> blobServiceClientSupplier.apply(location.withProperties(clientProperties)));
+ }
Review Comment:
<a href="#"><img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=7" align="top"></a> **Lambda parameter `endpoint` is shadowed and unused**
In the `computeIfAbsent` lambda, the parameter `endpoint` (the cache key, equal to `location.blobEndpoint`) is never referenced — the outer `location` variable is captured instead. The captured `location` happens to carry the same endpoint, so correctness is preserved, but the unused parameter is misleading.
```suggestion
private BlobServiceClient getBlobServiceClient(AzureLocation location) {
if (location.blobEndpoint.equals(lockBlobEndpoint)) {
return blobServiceClient;
}
AzureLocation locationWithProps = location.withProperties(clientProperties);
return secondaryBlobServiceClients.computeIfAbsent(
location.blobEndpoint,
endpoint -> blobServiceClientSupplier.apply(locationWithProps));
}
```
— *Greptile* ([original](https://github.com/yihua/hudi/pull/45#discussion_r3087235734)) (source:comment#3087235734)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add hudi azure based storage lock [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17951:
URL: https://github.com/apache/hudi/pull/17951#issuecomment-4272502938
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11235",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "43b35a0a65113c81463a237015cb99a18515d352",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11857",
"triggerID" : "43b35a0a65113c81463a237015cb99a18515d352",
"triggerType" : "PUSH"
}, {
"hash" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11938",
"triggerID" : "24d84063a6e9887312aedb0d1372d28c243a20f5",
"triggerType" : "PUSH"
}, {
"hash" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12280",
"triggerID" : "df7054b59932d78d931e443bc3ecbd1e4b9b83b4",
"triggerType" : "PUSH"
}, {
"hash" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12295",
"triggerID" : "2d956dc0beb807d556d51316f42d4728e41be82e",
"triggerType" : "PUSH"
}, {
"hash" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12505",
"triggerID" : "828646839bbe115c06e8a01e982e62ac4fc33292",
"triggerType" : "PUSH"
}, {
"hash" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12899",
"triggerID" : "598acc412f2dcbde52a06add8e314d67f80f3ee0",
"triggerType" : "PUSH"
}, {
"hash" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12932",
"triggerID" : "637a857b02bfb874dcc5eb5bfec909f94d271326",
"triggerType" : "PUSH"
}, {
"hash" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13266",
"triggerID" : "2ef504fe85cbb9a51e9f4f921843e7d37ec6a3f2",
"triggerType" : "PUSH"
}, {
"hash" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13347",
"triggerID" : "2992185d3b874b02c9c4ec77e6f96c8ee83770d3",
"triggerType" : "PUSH"
}, {
"hash" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350",
"triggerID" : "4107d19f79f42fa5e9c587e0e78936b7d8ca0ace",
"triggerType" : "PUSH"
}, {
"hash" : "d2d7cb2002a639a6aeedac2f31c35904fc45b2a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13353",
"triggerID" : "d2d7cb2002a639a6aeedac2f31c35904fc45b2a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4107d19f79f42fa5e9c587e0e78936b7d8ca0ace Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13350)
* d2d7cb2002a639a6aeedac2f31c35904fc45b2a8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13353)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org