You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "bhattmanish98 (via GitHub)" <gi...@apache.org> on 2025/01/02 06:59:31 UTC
[PR] HADOOP-19381: [ABFS] Support Rename and Delete operation over FNS-Blob endpoint [hadoop]
bhattmanish98 opened a new pull request, #7265:
URL: https://github.com/apache/hadoop/pull/7265
Currently, we only support rename and delete operations on the DFS endpoint. The reason for supporting rename and delete operations on the Blob endpoint is that the Blob endpoint does not account for hierarchy. We need to ensure that the HDFS contracts are maintained when performing rename and delete operations. Renaming or deleting a directory over the Blob endpoint requires the client to handle the orchestration and rename or delete all the blobs within the specified directory.
The task outlines the considerations for implementing rename and delete operations for the FNS-blob endpoint to ensure compatibility with HDFS contracts.
- **Blob Endpoint Usage**: The task addresses the need for abstraction in the code to maintain HDFS contracts while performing rename and delete operations on the blob endpoint, which does not support hierarchy.
- **Rename Operations**: The AzureBlobFileSystem#rename() method will use a RenameHandler instance to handle rename operations, with separate handlers for the DFS and blob endpoints. This method includes prechecks, destination adjustments, and orchestration of directory renaming for blobs.
- **Atomic Rename**: Atomic renaming is essential for blob endpoints, as it requires orchestration to copy or delete each blob within the directory. A configuration will allow developers to specify directories for atomic renaming, with a JSON file to track the status of renames.
- **Delete Operations**: Delete operations are simpler than renames, requiring fewer HDFS contract checks. For blob endpoints, the client must handle orchestration, including managing orphaned directories created by Az-copy.
- **Orchestration for Rename/Delete**: Orchestration for rename and delete operations over blob endpoints involves listing blobs and performing actions on each blob. The process must be optimized to handle large numbers of blobs efficiently.
- **Need for Optimization**: Optimization is crucial because the ListBlob API can return a maximum of 5000 blobs at once, necessitating multiple calls for large directories. The task proposes a producer-consumer model to handle blobs in parallel, thereby reducing processing time and memory usage.
- **Producer-Consumer Design**: The proposed design includes a producer to list blobs, a queue to store the blobs, and a consumer to process them in parallel. This approach aims to improve efficiency and mitigate memory issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: [ABFS] Support Rename and Delete operation over FNS-Blob endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2569169036
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 37s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 47s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 8 unchanged - 3 fixed = 8 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 18s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 40s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 135m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7218cddf04e8 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0377a7e2b76152549527eb02dd827009735deca3 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/3/testReport/ |
| Max. process+thread count | 532 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1912738850
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -499,21 +556,50 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws IOException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
- final String destination,
- final String continuation,
- final TracingContext tracingContext,
- final String sourceEtag,
- final boolean isMetadataIncompleteState,
- final boolean isNamespaceEnabled) throws IOException {
- /**
- * TODO: [FnsOverBlob] To be implemented as part of rename-delete over blob endpoint work. <a href="https://issues.apache.org/jira/browse/HADOOP-19233">HADOOP-19233</a>.
- */
- throw new NotImplementedException("Rename operation on Blob endpoint yet to be implemented.");
+ final String destination,
+ final String continuation,
+ final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
+ throws IOException {
+ BlobRenameHandler blobRenameHandler = getBlobRenameHandler(source,
+ destination, sourceEtag, isAtomicRenameKey(source), tracingContext
+ );
+ incrementAbfsRenamePath();
+ if (blobRenameHandler.execute()) {
+ final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
+ final URL url = createRequestUrl(destination, abfsUriQueryBuilder.toString());
+ final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
+ final AbfsRestOperation successOp = getAbfsRestOperation(
+ AbfsRestOperationType.RenamePath, HTTP_METHOD_PUT,
+ url, requestHeaders);
+ successOp.hardSetResult(HttpURLConnection.HTTP_OK);
+ return new AbfsClientRenameResult(successOp, true, false);
+ } else {
+ throw new AbfsRestOperationException(HTTP_INTERNAL_ERROR,
+ AzureServiceErrorCode.UNKNOWN.getErrorCode(),
+ "FNS-Blob Rename was not successfull",
Review Comment:
typo: successful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936108534
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -782,10 +790,10 @@ public synchronized void close() throws IOException {
}
if (abfsStore.getClient().isMetricCollectionEnabled()) {
TracingContext tracingMetricContext = new TracingContext(
- clientCorrelationId,
- fileSystemId, FSOperationType.GET_ATTR, true,
- tracingHeaderFormat,
- listener, abfsCounters.toString());
+ clientCorrelationId,
Review Comment:
undo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936989900
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java:
##########
@@ -409,6 +412,24 @@ public AbfsRestOperation createPath(final String path,
return op;
}
+ public void createNonRecursivePreCheck(Path parentPath,
Review Comment:
inherit doc annotation is missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931752429
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931757740
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -359,6 +375,34 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
return op;
}
+ private void fixAtomicEntriesInListResults(final AbfsRestOperation op,
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2624608093
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 11m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 6s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 7s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/16/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 14 unchanged - 3 fixed = 15 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 56s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 126m 52s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 6429d43b7576 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2166246fcdcfc4c1b71a5d4e766a35da3507697b |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/16/testReport/ |
| Max. process+thread count | 546 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936764773
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -689,21 +789,53 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws IOException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
final String destination,
final String continuation,
final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
Review Comment:
as discussed, done the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933332401
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -415,18 +415,19 @@ public FSDataOutputStream createNonRecursive(final Path f, final FsPermission pe
ERR_CREATE_ON_ROOT,
null);
}
- final Path parent = f.getParent();
+
TracingContext tracingContext = new TracingContext(clientCorrelationId,
fileSystemId, FSOperationType.CREATE_NON_RECURSIVE, tracingHeaderFormat,
listener);
- final FileStatus parentFileStatus = tryGetFileStatus(parent, tracingContext);
-
- if (parentFileStatus == null) {
- throw new FileNotFoundException("Cannot create file "
- + f.getName() + " because parent folder does not exist.");
+ try {
+ Path qualifiedPath = makeQualified(f);
+ getAbfsStore().createNonRecursivePreCheck(qualifiedPath, tracingContext);
+ return create(f, permission, overwrite, bufferSize, replication,
+ blockSize, progress);
+ } catch (AzureBlobFileSystemException ex) {
+ checkException(f, ex);
Review Comment:
checkException() was not happening earlier. This might cause difference in behavior?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1124,8 +1144,11 @@ public void delete(final Path path, final boolean recursive,
try (AbfsPerfInfo perfInfo = startTracking("delete", "deletePath")) {
AbfsRestOperation op = getClient().deletePath(relativePath, recursive,
continuation, tracingContext, getIsNamespaceEnabled(tracingContext));
- perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ if (op != null) {
Review Comment:
Same here. Why this change is needed.?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobDeleteHandler.java:
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_CONFLICT;
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.NON_EMPTY_DIRECTORY_DELETE;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for delete over Blob endpoint. Blob endpoint for flat-namespace
+ * account does not support directory delete. This class is responsible for
+ * deleting the blobs and creating the parent directory marker file if needed.
+ */
+public class BlobDeleteHandler extends ListActionTaker {
+
+ private final Path path;
+
+ private final boolean recursive;
+
+ private boolean nonRecursiveDeleteDirectoryFailed = false;
+
+ private final TracingContext tracingContext;
+
+ private final AtomicInteger deleteCount = new AtomicInteger(0);
+
+ /** Constructor
+ *
+ * @param path path to delete.
+ * @param recursive if true, delete the path recursively.
+ * @param abfsBlobClient client to use for blob operations.
+ * @param tracingContext tracing context.
+ */
+ public BlobDeleteHandler(final Path path,
+ final boolean recursive,
+ final AbfsBlobClient abfsBlobClient,
+ final TracingContext tracingContext) {
+ super(path, abfsBlobClient, tracingContext);
+ this.path = path;
+ this.recursive = recursive;
+ this.tracingContext = tracingContext;
+ }
+
+ /**{@inheritDoc}
+ *
+ * @return the maximum number of parallelism for delete operation.
+ */
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobDeleteDirConsumptionParallelism();
+ }
+
+ /** Delete the path.
+ *
+ * @param path path to delete.
+ * @return true if the path is deleted.
+ * @throws AzureBlobFileSystemException server error.
+ */
+ private boolean deleteInternal(final Path path)
+ throws AzureBlobFileSystemException {
+ getAbfsClient().deleteBlobPath(path, null, tracingContext);
Review Comment:
`deleteBlobPath` return AbfsRestOperation but it is never used. May be we can update it to return void.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1063,11 +1085,6 @@ public boolean rename(final Path source,
long countAggregate = 0;
boolean shouldContinue;
- if (isAtomicRenameKey(source.getName())) {
Review Comment:
Why we are removing this Warning?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -68,9 +70,14 @@
import org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider;
import org.apache.hadoop.fs.azurebfs.security.ContextEncryptionAdapter;
import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.apache.hadoop.classification.VisibleForTesting;
Review Comment:
fix import ordering
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -909,23 +1025,46 @@ public AbfsRestOperation read(final String path,
}
/**
- * Orchestration for delete operation to be implemented.
+ * Get Rest Operation for API <a href = https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete></a>.
Review Comment:
This is not an API implementation, javadoc should say Orchestration only.
This will be needed for API implementation of delete path
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1089,8 +1106,11 @@ public boolean rename(final Path source,
isNamespaceEnabled);
AbfsRestOperation op = abfsClientRenameResult.getOp();
- perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ if (op != null) {
Review Comment:
Why will op be null now?
Why this change needed?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java:
##########
@@ -290,5 +290,13 @@ public static ApiVersion getCurrentVersion() {
public static final String JDK_FALLBACK = "JDK_fallback";
public static final String KEEP_ALIVE_CACHE_CLOSED = "KeepAliveCache is closed";
+ public static final String COPY_STATUS_SUCCESS = "success";
+ public static final String COPY_STATUS_PENDING = "pending";
+ public static final String COPY_STATUS_ABORTED = "aborted";
+ public static final String COPY_STATUS_FAILED = "failed";
+
+ public static final String ATOMIC_DIR_RENAME_RECOVERY_ON_GET_PATH_EXCEPTION =
Review Comment:
This does not seem like a Http Constant. Its a error message move it to `AbfsErrors`
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -38,6 +38,8 @@
import java.util.Map;
import java.util.TreeMap;
import java.util.UUID;
+import java.util.Arrays;
Review Comment:
fix Import ordering
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -499,21 +578,50 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws IOException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
- final String destination,
- final String continuation,
- final TracingContext tracingContext,
- final String sourceEtag,
- final boolean isMetadataIncompleteState,
- final boolean isNamespaceEnabled) throws IOException {
- /**
- * TODO: [FnsOverBlob] To be implemented as part of rename-delete over blob endpoint work. <a href="https://issues.apache.org/jira/browse/HADOOP-19233">HADOOP-19233</a>.
- */
- throw new NotImplementedException("Rename operation on Blob endpoint yet to be implemented.");
+ final String destination,
Review Comment:
We should keep same indentation format throughout the file.
Please fix it for all methods added
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java:
##########
@@ -45,10 +46,10 @@
* AbfsLease manages an Azure blob lease. It acquires an infinite lease on instantiation and
* releases the lease when free() is called. Use it to prevent writes to the blob by other
* processes that don't have the lease.
- *
+ * <p>
Review Comment:
Don't we need closing tags for `</p>`
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -22,6 +22,7 @@
import java.io.IOException;
import java.io.InputStream;
import java.io.UnsupportedEncodingException;
+import java.io.FileNotFoundException;
Review Comment:
fix import ordering
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936111184
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1118,11 +1135,14 @@ public boolean rename(final Path source,
final AbfsClientRenameResult abfsClientRenameResult =
getClient().renamePath(sourceRelativePath, destinationRelativePath,
continuation, tracingContext, sourceEtag, false,
- isNamespaceEnabled);
+ isNamespaceEnabled);
Review Comment:
remove isNamespace enabled as param from rename and delete path as that info is available as a client method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936992194
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsPermission.java:
##########
@@ -68,7 +69,8 @@ public boolean equals(Object obj) {
* @return a permission object for the provided string representation
*/
public static AbfsPermission valueOf(final String abfsSymbolicPermission) {
- if (abfsSymbolicPermission == null) {
+ if (abfsSymbolicPermission == null
Review Comment:
isEmpty will cover the null check also
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsPermission.java:
##########
@@ -100,7 +102,8 @@ public static AbfsPermission valueOf(final String abfsSymbolicPermission) {
* extended ACL; otherwise false.
*/
public static boolean isExtendedAcl(final String abfsSymbolicPermission) {
- if (abfsSymbolicPermission == null) {
+ if (abfsSymbolicPermission == null
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931699666
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
+ getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936664518
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1089,8 +1106,11 @@ public boolean rename(final Path source,
isNamespaceEnabled);
AbfsRestOperation op = abfsClientRenameResult.getOp();
- perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ if (op != null) {
Review Comment:
I recall that we decided that its not needed and op will never be null and this additional diff can be avoided.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1124,8 +1144,11 @@ public void delete(final Path path, final boolean recursive,
try (AbfsPerfInfo perfInfo = startTracking("delete", "deletePath")) {
AbfsRestOperation op = getClient().deletePath(relativePath, recursive,
continuation, tracingContext, getIsNamespaceEnabled(tracingContext));
- perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ if (op != null) {
Review Comment:
Same reply as rename
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933420117
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -22,6 +22,7 @@
import java.io.IOException;
import java.io.InputStream;
import java.io.UnsupportedEncodingException;
+import java.io.FileNotFoundException;
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933386297
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1089,8 +1106,11 @@ public boolean rename(final Path source,
isNamespaceEnabled);
AbfsRestOperation op = abfsClientRenameResult.getOp();
- perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ if (op != null) {
Review Comment:
This was discussed offline that we will keep this null check here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931691647
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java:
##########
@@ -314,4 +347,278 @@ public void deleteBlobDirParallelThreadToDeleteOnDifferentTracingContext()
fs.delete(new Path("/testDir"), true);
fs.close();
}
+
+ private void assumeBlobClient() throws IOException {
Review Comment:
getAbfsServiceType() method is not there in trunk. Keeping it as it is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931756612
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/AzureServiceErrorCode.java:
##########
@@ -45,6 +45,8 @@ public enum AzureServiceErrorCode {
INVALID_SOURCE_OR_DESTINATION_RESOURCE_TYPE("InvalidSourceOrDestinationResourceType", HttpURLConnection.HTTP_CONFLICT, null),
RENAME_DESTINATION_PARENT_PATH_NOT_FOUND("RenameDestinationParentPathNotFound", HttpURLConnection.HTTP_NOT_FOUND, null),
INVALID_RENAME_SOURCE_PATH("InvalidRenameSourcePath", HttpURLConnection.HTTP_CONFLICT, null),
+ DIRECTORY_NOT_EMPTY_DELETE("DirectoryNotEmpty", HttpURLConnection.HTTP_CONFLICT,
Review Comment:
Renamed it to NON_EMPTY_DIRECTORY_DELETE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1938977495
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ListBlobQueue.java:
##########
@@ -170,7 +170,7 @@ private List<Path> dequeue() {
return pathListForConsumption;
}
- private synchronized int size() {
+ synchronized int size() {
Review Comment:
This method is called from TestListActionTaker, thats why needed this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937087835
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1155,9 +1169,10 @@ public void delete(final Path path, final boolean recursive,
do {
try (AbfsPerfInfo perfInfo = startTracking("delete", "deletePath")) {
AbfsRestOperation op = getClient().deletePath(relativePath, recursive,
- continuation, tracingContext, getIsNamespaceEnabled(tracingContext));
+ continuation, tracingContext);
perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ continuation = op.getResult()
Review Comment:
Avoid diffs
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1144,9 +1158,9 @@ public void delete(final Path path, final boolean recursive,
boolean shouldContinue = true;
LOG.debug("delete filesystem: {} path: {} recursive: {}",
- getClient().getFileSystem(),
- path,
- String.valueOf(recursive));
+ getClient().getFileSystem(),
Review Comment:
Avoid diffs
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -673,7 +694,7 @@ public OutputStream createFile(final Path path,
}
final ContextEncryptionAdapter contextEncryptionAdapter;
- if (createClient.getEncryptionType() == EncryptionType.ENCRYPTION_CONTEXT) {
+ if (getClient().getEncryptionType() == EncryptionType.ENCRYPTION_CONTEXT) {
Review Comment:
Why are we changing this.
This seems wrong. Right @anmolanmol1234
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1114,15 +1129,14 @@ public boolean rename(final Path source,
do {
try (AbfsPerfInfo perfInfo = startTracking("rename", "renamePath")) {
- boolean isNamespaceEnabled = getIsNamespaceEnabled(tracingContext);
final AbfsClientRenameResult abfsClientRenameResult =
getClient().renamePath(sourceRelativePath, destinationRelativePath,
- continuation, tracingContext, sourceEtag, false,
- isNamespaceEnabled);
+ continuation, tracingContext, sourceEtag, false);
AbfsRestOperation op = abfsClientRenameResult.getOp();
perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ continuation = op.getResult()
Review Comment:
Avoid diffs
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1694,16 +1976,24 @@ private boolean isNonEmptyListing(String path,
* @return True if empty results without continuation token.
*/
private boolean isEmptyListResults(AbfsHttpOperation result) {
Review Comment:
Why diffs in this method, please avoid unneceessary diffs.
I agree that they don't break anything but the PR is already big. It should not touch the code it is not supposed to do.
For general improvements please raise another Jira and PR.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -18,6 +18,7 @@
package org.apache.hadoop.fs.azurebfs;
+import javax.annotation.Nullable;
Review Comment:
Import ordering looks wrong. Causing unneccessary diff as well.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -42,25 +42,42 @@
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
+import org.slf4j.Logger;
Review Comment:
Thanks for fixing import ordering in this classs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937107764
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -18,6 +18,7 @@
package org.apache.hadoop.fs.azurebfs;
+import javax.annotation.Nullable;
Review Comment:
These ordering are automatically picked based on the XML format we use to code formatting. Do I need to revert them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1938968273
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -649,18 +647,16 @@ public void deleteFilesystem(TracingContext tracingContext)
*
* @throws IOException if there is an issue with the operation.
*/
- public void createNonRecursivePreCheck(final Path path,
+ public OutputStream createNonRecursive(final Path path,
+ final FileSystem.Statistics statistics, final boolean overwrite,
+ final FsPermission permission, final FsPermission umask,
TracingContext tracingContext)
throws IOException {
- if (path.getParent() != null) {
- getClient().createNonRecursivePreCheck(path.getParent(),
- tracingContext);
- } else {
- throw new AbfsRestOperationException(HTTP_CONFLICT,
- AzureServiceErrorCode.PATH_CONFLICT.getErrorCode(),
- ERR_CREATE_ON_ROOT,
- null);
- }
+ getClient().createNonRecursivePreCheck(path.getParent(),
Review Comment:
debug logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2627473406
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 14 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 115m 52s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/27/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux c85b73176292 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f3acefb9271a234c88935085c7fc1be1145c8457 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/27/testReport/ |
| Max. process+thread count | 700 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/27/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2624048685
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 23s | | https://github.com/apache/hadoop/pull/7265 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| JIRA Issue | HADOOP-19233 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/13/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1912753857
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobDeleteHandler.java:
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_CONFLICT;
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.DIRECTORY_NOT_EMPTY_DELETE;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for delete over Blob endpoint. Blob endpoint for flat-namespace
+ * account does not support director delete. This class is responsible for
+ * deleting the blobs and creating the parent directory marker file if needed.
+ */
+public class BlobDeleteHandler extends ListActionTaker {
+
+ private final Path path;
+
+ private final boolean recursive;
+
+ private boolean nonRecursiveDeleteDirectoryFailed = false;
+
+ private final TracingContext tracingContext;
+
+ private final AtomicInteger deleteCount = new AtomicInteger(0);
+
+ /** Constructor
+ *
+ * @param path path to delete.
+ * @param recursive if true, delete the path recursively.
+ * @param abfsBlobClient client to use for blob operations.
+ * @param tracingContext tracing context.
+ */
+ public BlobDeleteHandler(final Path path,
+ final boolean recursive,
+ final AbfsBlobClient abfsBlobClient,
+ final TracingContext tracingContext) {
+ super(path, abfsBlobClient, tracingContext);
+ this.path = path;
+ this.recursive = recursive;
+ this.tracingContext = tracingContext;
+ }
+
+ /**{@inheritDoc}
+ *
+ * @return the maximum number of parallelism for delete operation.
+ */
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobDeleteDirConsumptionParallelism();
+ }
+
+ /** Delete the path.
+ *
+ * @param path path to delete.
+ * @return true if the path is deleted.
+ * @throws AzureBlobFileSystemException server error.
+ */
+ private boolean deleteInternal(final Path path)
+ throws AzureBlobFileSystemException {
+ getAbfsClient().deleteBlobPath(path, null, tracingContext);
+ deleteCount.incrementAndGet();
+ return true;
+ }
+
+ /**
+ * Orchestrate the delete operation.
+ *
+ * @return true if the delete operation is successful.
+ * @throws AzureBlobFileSystemException if deletion fails due to server error or path doesn't exist.
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ /*
+ * ABFS is not aware if it's a file or directory. So, we need to list the
+ * path and delete the listed objects. The listing returns the children of
+ * the path and not the path itself.
+ */
+ listRecursiveAndTakeAction();
+ if (nonRecursiveDeleteDirectoryFailed) {
+ throw new AbfsRestOperationException(HTTP_CONFLICT,
+ DIRECTORY_NOT_EMPTY_DELETE.getErrorCode(),
+ DIRECTORY_NOT_EMPTY_DELETE.getErrorMessage(),
+ new PathIOException(path.toString(),
+ "Non-recursive delete of non-empty directory"));
+ }
+ tracingContext.setOperatedBlobCount(deleteCount.get() + 1);
+ /*
+ * If path is actually deleted.
+ */
+ boolean deleted;
+ try {
+ /*
+ * Delete the required path.
+ * Directory needs to be safely delete the path, as the path can be implicit.
Review Comment:
typo: grammar issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917796817
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -499,21 +526,36 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws AzureBlobFileSystemException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
- final String destination,
- final String continuation,
- final TracingContext tracingContext,
- final String sourceEtag,
- final boolean isMetadataIncompleteState,
- final boolean isNamespaceEnabled) throws IOException {
- /**
- * TODO: [FnsOverBlob] To be implemented as part of rename-delete over blob endpoint work. <a href="https://issues.apache.org/jira/browse/HADOOP-19233">HADOOP-19233</a>.
- */
- throw new NotImplementedException("Rename operation on Blob endpoint yet to be implemented.");
+ final String destination,
+ final String continuation,
+ final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
+ throws IOException {
+ BlobRenameHandler blobRenameHandler = getBlobRenameHandler(source,
+ destination, sourceEtag, isAtomicRenameKey(source), tracingContext
+ );
+ incrementAbfsRenamePath();
+ return blobRenameHandler.execute();
Review Comment:
As per the offline discussion, made the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916517959
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
+//import java.util.Collections;
+//import java.util.List;
+import java.util.Random;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsDriverException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.contracts.services.BlobAppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static java.net.HttpURLConnection.HTTP_PRECON_FAILED;
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.BLOCK_ID_LENGTH;
+//import static org.apache.hadoop.fs.azurebfs.services.AzureIngressHandler.generateBlockListXml;
+
+/**
+ * For a directory enabled for atomic-rename, before rename starts, a file with
+ * -RenamePending.json suffix is created. In this file, the states required for the
+ * rename operation are given. This file is created by {@link #preRename()} method.
+ * This is important in case the JVM process crashes during rename, the atomicity
+ * will be maintained, when the job calls {@link AzureBlobFileSystem#listStatus(Path)}
+ * or {@link AzureBlobFileSystem#getFileStatus(Path)}. On these API calls to filesystem,
+ * it will be checked if there is any RenamePending JSON file. If yes, the crashed rename
+ * operation would be resumed as per the file.
+ */
+public class RenameAtomicity {
+
+ private final TracingContext tracingContext;
+
+ private Path src, dst;
+
+ private String srcEtag;
+
+ private final AbfsBlobClient abfsClient;
+
+ private final Path renameJsonPath;
+
+ public static final String SUFFIX = "-RenamePending.json";
+
+ private int preRenameRetryCount = 0;
+
+ private int renamePendingJsonLen;
+
+ private final AbfsLease sourcePathLease;
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private static final Random RANDOM = new Random();
+
+ /**
+ * Performs pre-rename operations. Creates a file with -RenamePending.json
+ * suffix in the source parent directory. This file contains the states
+ * required for the rename operation.
+ *
+ * @param src Source path
+ * @param dst Destination path
+ * @param renameJsonPath Path of the JSON file to be created
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ */
+ public RenameAtomicity(final Path src, final Path dst,
+ final Path renameJsonPath,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient) {
+ this.src = src;
+ this.dst = dst;
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.sourcePathLease = null;
+ }
+
+ /**
+ * Resumes the rename operation from the JSON file.
+ *
+ * @param renameJsonPath Path of the JSON file
+ * @param renamePendingJsonFileLen Length of the JSON file
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ * @param sourceLease Lease of the source directory
+ */
+ public RenameAtomicity(final Path renameJsonPath,
+ final int renamePendingJsonFileLen,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient,
+ final AbfsLease sourceLease) {
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.renamePendingJsonLen = renamePendingJsonFileLen;
+ this.sourcePathLease = sourceLease;
+ }
+
+ /**
+ * Redo the rename operation from the JSON file.
+ *
+ * @throws AzureBlobFileSystemException If the redo operation fails.
+ */
+ public void redo() throws AzureBlobFileSystemException {
+ byte[] buffer = readRenamePendingJson(renameJsonPath, renamePendingJsonLen);
+ String contents = new String(buffer, Charset.defaultCharset());
+ try {
+ final RenamePendingJsonFormat renamePendingJsonFormatObj;
+ try {
+ renamePendingJsonFormatObj = objectMapper.readValue(contents,
+ RenamePendingJsonFormat.class);
+ } catch (JsonProcessingException e) {
+ return;
Review Comment:
Is it right to just return without throwing an exception ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916521221
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
+//import java.util.Collections;
+//import java.util.List;
+import java.util.Random;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsDriverException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.contracts.services.BlobAppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static java.net.HttpURLConnection.HTTP_PRECON_FAILED;
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.BLOCK_ID_LENGTH;
+//import static org.apache.hadoop.fs.azurebfs.services.AzureIngressHandler.generateBlockListXml;
+
+/**
+ * For a directory enabled for atomic-rename, before rename starts, a file with
+ * -RenamePending.json suffix is created. In this file, the states required for the
+ * rename operation are given. This file is created by {@link #preRename()} method.
+ * This is important in case the JVM process crashes during rename, the atomicity
+ * will be maintained, when the job calls {@link AzureBlobFileSystem#listStatus(Path)}
+ * or {@link AzureBlobFileSystem#getFileStatus(Path)}. On these API calls to filesystem,
+ * it will be checked if there is any RenamePending JSON file. If yes, the crashed rename
+ * operation would be resumed as per the file.
+ */
+public class RenameAtomicity {
+
+ private final TracingContext tracingContext;
+
+ private Path src, dst;
+
+ private String srcEtag;
+
+ private final AbfsBlobClient abfsClient;
+
+ private final Path renameJsonPath;
+
+ public static final String SUFFIX = "-RenamePending.json";
+
+ private int preRenameRetryCount = 0;
+
+ private int renamePendingJsonLen;
+
+ private final AbfsLease sourcePathLease;
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private static final Random RANDOM = new Random();
+
+ /**
+ * Performs pre-rename operations. Creates a file with -RenamePending.json
+ * suffix in the source parent directory. This file contains the states
+ * required for the rename operation.
+ *
+ * @param src Source path
+ * @param dst Destination path
+ * @param renameJsonPath Path of the JSON file to be created
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ */
+ public RenameAtomicity(final Path src, final Path dst,
+ final Path renameJsonPath,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient) {
+ this.src = src;
+ this.dst = dst;
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.sourcePathLease = null;
+ }
+
+ /**
+ * Resumes the rename operation from the JSON file.
+ *
+ * @param renameJsonPath Path of the JSON file
+ * @param renamePendingJsonFileLen Length of the JSON file
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ * @param sourceLease Lease of the source directory
+ */
+ public RenameAtomicity(final Path renameJsonPath,
+ final int renamePendingJsonFileLen,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient,
+ final AbfsLease sourceLease) {
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.renamePendingJsonLen = renamePendingJsonFileLen;
+ this.sourcePathLease = sourceLease;
+ }
+
+ /**
+ * Redo the rename operation from the JSON file.
+ *
+ * @throws AzureBlobFileSystemException If the redo operation fails.
+ */
+ public void redo() throws AzureBlobFileSystemException {
+ byte[] buffer = readRenamePendingJson(renameJsonPath, renamePendingJsonLen);
+ String contents = new String(buffer, Charset.defaultCharset());
+ try {
+ final RenamePendingJsonFormat renamePendingJsonFormatObj;
+ try {
+ renamePendingJsonFormatObj = objectMapper.readValue(contents,
+ RenamePendingJsonFormat.class);
+ } catch (JsonProcessingException e) {
+ return;
+ }
+ if (renamePendingJsonFormatObj != null && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getOldFolderName())
+ && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getNewFolderName())
+ && StringUtils.isNotEmpty(renamePendingJsonFormatObj.getETag())) {
+ this.src = new Path(renamePendingJsonFormatObj.getOldFolderName());
+ this.dst = new Path(renamePendingJsonFormatObj.getNewFolderName());
+ this.srcEtag = renamePendingJsonFormatObj.getETag();
+
+ BlobRenameHandler blobRenameHandler = new BlobRenameHandler(
+ this.src.toUri().getPath(), dst.toUri().getPath(),
+ abfsClient, srcEtag, true, true,
+ sourcePathLease, tracingContext);
+
+ blobRenameHandler.execute();
+ }
+ } finally {
+ deleteRenamePendingJson();
+ }
+ }
+
+ /** Read the JSON file.
+ *
+ * @param path Path of the JSON file
+ * @param len Length of the JSON file
+ * @return Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the read operation fails.
+ */
+ @VisibleForTesting
+ byte[] readRenamePendingJson(Path path, int len)
+ throws AzureBlobFileSystemException {
+ byte[] bytes = new byte[len];
+ abfsClient.read(path.toUri().getPath(), 0, bytes, 0,
+ len, null, null, null,
+ tracingContext);
+ return bytes;
+ }
+
+ /** Generate a random block ID.
+ *
+ * @return Random block ID
+ */
+ public static String generateBlockId() {
+ // PutBlock on the path.
+ byte[] blockIdByteArray = new byte[BLOCK_ID_LENGTH];
+ RANDOM.nextBytes(blockIdByteArray);
+ return new String(Base64.encodeBase64(blockIdByteArray),
+ StandardCharsets.UTF_8);
+ }
+
+ /** Create the JSON file with the contents.
+ *
+ * @param path Path of the JSON file
+ * @param bytes Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the create operation fails.
+ */
+ @VisibleForTesting
+ void createRenamePendingJson(Path path, byte[] bytes)
+ throws AzureBlobFileSystemException {
+ // PutBlob on the path.
+ AbfsRestOperation putBlobOp = abfsClient.createPath(path.toUri().getPath(),
+ true,
+ true,
+ null,
+ false,
+ null,
+ null,
+ tracingContext);
+ String eTag = extractEtagHeader(putBlobOp.getResult());
+
+ String blockId = generateBlockId();
+ AppendRequestParameters appendRequestParameters
+ = new AppendRequestParameters(0, 0,
+ bytes.length, AppendRequestParameters.Mode.APPEND_MODE, false, null,
+ abfsClient.getAbfsConfiguration().isExpectHeaderEnabled(),
+ new BlobAppendRequestParameters(blockId, eTag));
+
+ abfsClient.append(path.toUri().getPath(), bytes,
+ appendRequestParameters, null, null, tracingContext);
+
+// List<String> blockIdList = new ArrayList<>(Collections.singleton(blockId));
Review Comment:
Remove commented code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916403336
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
+ getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
+ == BlobCopyProgress.PENDING) {
+ try {
+ Thread.sleep(pollWait);
+ } catch (InterruptedException ignored) {
+
+ }
+ }
+ }
+
+ /** Gets the source path of the copy operation.
+ *
+ * @param dstPathStatus server response for the GetBlobProperties API on the
+ * destination path.
+ *
+ * @return source path of the copy operation
+ */
+ private String getDstSource(final AbfsRestOperation dstPathStatus) {
+ try {
+ String responseHeader = dstPathStatus.getResult()
+ .getResponseHeader(X_MS_COPY_SOURCE);
+ if (responseHeader == null) {
+ return null;
+ }
+ return new URL(responseHeader).toURI().getPath();
+ } catch (URISyntaxException | MalformedURLException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Verifies if the blob copy is success or a failure or still in progress.
+ *
+ * @param dstPath path of the destination for the copying
+ * @param tracingContext object of tracingContext used for the tracing of the
+ * server calls.
+ * @param copyId id returned by server on the copy server-call. This id gets
+ * attached to blob and is returned by GetBlobProperties API on the destination.
+ *
+ * @return BlobCopyProgress indicating the status of the copy operation
+ *
+ * @throws AzureBlobFileSystemException exception returned in making server call
+ * for GetBlobProperties on the path. It can be thrown if the copyStatus is failure
+ * or is aborted.
+ */
+ @VisibleForTesting
+ public BlobCopyProgress handleCopyInProgress(final Path dstPath,
+ final TracingContext tracingContext,
+ final String copyId) throws AzureBlobFileSystemException {
+ AbfsRestOperation op = getAbfsClient().getPathStatus(dstPath.toUri().getPath(),
+ tracingContext, null, false);
+
+ if (op.getResult() != null && copyId.equals(
+ op.getResult().getResponseHeader(X_MS_COPY_ID))) {
+ final String copyStatus = op.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(copyStatus)) {
+ return BlobCopyProgress.SUCCESS;
+ }
+ if (COPY_STATUS_FAILED.equalsIgnoreCase(copyStatus)) {
+ throw new AbfsRestOperationException(
+ COPY_BLOB_FAILED.getStatusCode(), COPY_BLOB_FAILED.getErrorCode(),
+ String.format("copy to path %s failed due to: %s",
+ dstPath.toUri().getPath(),
+ op.getResult().getResponseHeader(X_MS_COPY_STATUS_DESCRIPTION)),
+ new Exception(COPY_BLOB_FAILED.getErrorCode()));
+ }
+ if (COPY_STATUS_ABORTED.equalsIgnoreCase(copyStatus)) {
+ throw new AbfsRestOperationException(
+ COPY_BLOB_ABORTED.getStatusCode(), COPY_BLOB_ABORTED.getErrorCode(),
+ String.format("copy to path %s aborted", dstPath.toUri().getPath()),
+ new Exception(COPY_BLOB_ABORTED.getErrorCode()));
+ }
+ }
+ return BlobCopyProgress.PENDING;
+ }
+
+ /**
+ * Translates the destination path for a blob part of a source directory getting
+ * renamed.
+ *
+ * @param destinationDir destination directory for the rename operation
+ * @param blobPath path of blob inside sourceDir being renamed.
+ * @param sourceDir source directory for the rename operation
+ *
+ * @return translated path for the blob
+ */
+ private Path createDestinationPathForBlobPartOfRenameSrcDir(final Path destinationDir,
+ final Path blobPath, final Path sourceDir) {
+ String destinationPathStr = destinationDir.toUri().getPath();
+ String sourcePathStr = sourceDir.toUri().getPath();
+ String srcBlobPropertyPathStr = blobPath.toUri().getPath();
+ if (sourcePathStr.equals(srcBlobPropertyPathStr)) {
+ return destinationDir;
+ }
+ return new Path(
+ destinationPathStr + ROOT_PATH + srcBlobPropertyPathStr.substring(
+ sourcePathStr.length()));
+ }
+
+ /** Get information of the path.
+ *
+ * @param path path for which the path information is to be fetched
+ * @param tracingContext object of tracingContext used for the tracing of the
+ * server calls.
+ *
+ * @return object containing the path information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private PathInformation getPathInformation(Path path,
+ TracingContext tracingContext)
+ throws AzureBlobFileSystemException {
+ try {
+ AbfsRestOperation op = getAbfsClient().getPathStatus(path.toString(),
+ tracingContext, null, true);
+
+ return new PathInformation(true,
+ getAbfsClient().checkIsDir(op.getResult()),
+ extractEtagHeader(op.getResult()),
+ op.getResult() instanceof AbfsHttpOperation.AbfsHttpOperationWithFixedResultForGetFileStatus);
+ } catch (AzureBlobFileSystemException e) {
Review Comment:
better to catch AbfsRestOperation itself ?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
+ getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
+ == BlobCopyProgress.PENDING) {
+ try {
+ Thread.sleep(pollWait);
+ } catch (InterruptedException ignored) {
+
+ }
+ }
+ }
+
+ /** Gets the source path of the copy operation.
+ *
+ * @param dstPathStatus server response for the GetBlobProperties API on the
+ * destination path.
+ *
+ * @return source path of the copy operation
+ */
+ private String getDstSource(final AbfsRestOperation dstPathStatus) {
+ try {
+ String responseHeader = dstPathStatus.getResult()
+ .getResponseHeader(X_MS_COPY_SOURCE);
+ if (responseHeader == null) {
+ return null;
+ }
+ return new URL(responseHeader).toURI().getPath();
+ } catch (URISyntaxException | MalformedURLException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Verifies if the blob copy is success or a failure or still in progress.
+ *
+ * @param dstPath path of the destination for the copying
+ * @param tracingContext object of tracingContext used for the tracing of the
+ * server calls.
+ * @param copyId id returned by server on the copy server-call. This id gets
+ * attached to blob and is returned by GetBlobProperties API on the destination.
+ *
+ * @return BlobCopyProgress indicating the status of the copy operation
+ *
+ * @throws AzureBlobFileSystemException exception returned in making server call
+ * for GetBlobProperties on the path. It can be thrown if the copyStatus is failure
+ * or is aborted.
+ */
+ @VisibleForTesting
+ public BlobCopyProgress handleCopyInProgress(final Path dstPath,
+ final TracingContext tracingContext,
+ final String copyId) throws AzureBlobFileSystemException {
+ AbfsRestOperation op = getAbfsClient().getPathStatus(dstPath.toUri().getPath(),
+ tracingContext, null, false);
+
+ if (op.getResult() != null && copyId.equals(
+ op.getResult().getResponseHeader(X_MS_COPY_ID))) {
+ final String copyStatus = op.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(copyStatus)) {
+ return BlobCopyProgress.SUCCESS;
+ }
+ if (COPY_STATUS_FAILED.equalsIgnoreCase(copyStatus)) {
+ throw new AbfsRestOperationException(
+ COPY_BLOB_FAILED.getStatusCode(), COPY_BLOB_FAILED.getErrorCode(),
+ String.format("copy to path %s failed due to: %s",
+ dstPath.toUri().getPath(),
+ op.getResult().getResponseHeader(X_MS_COPY_STATUS_DESCRIPTION)),
+ new Exception(COPY_BLOB_FAILED.getErrorCode()));
+ }
+ if (COPY_STATUS_ABORTED.equalsIgnoreCase(copyStatus)) {
+ throw new AbfsRestOperationException(
+ COPY_BLOB_ABORTED.getStatusCode(), COPY_BLOB_ABORTED.getErrorCode(),
+ String.format("copy to path %s aborted", dstPath.toUri().getPath()),
+ new Exception(COPY_BLOB_ABORTED.getErrorCode()));
+ }
+ }
+ return BlobCopyProgress.PENDING;
+ }
+
+ /**
+ * Translates the destination path for a blob part of a source directory getting
+ * renamed.
+ *
+ * @param destinationDir destination directory for the rename operation
+ * @param blobPath path of blob inside sourceDir being renamed.
+ * @param sourceDir source directory for the rename operation
+ *
+ * @return translated path for the blob
+ */
+ private Path createDestinationPathForBlobPartOfRenameSrcDir(final Path destinationDir,
+ final Path blobPath, final Path sourceDir) {
+ String destinationPathStr = destinationDir.toUri().getPath();
+ String sourcePathStr = sourceDir.toUri().getPath();
+ String srcBlobPropertyPathStr = blobPath.toUri().getPath();
+ if (sourcePathStr.equals(srcBlobPropertyPathStr)) {
+ return destinationDir;
+ }
+ return new Path(
+ destinationPathStr + ROOT_PATH + srcBlobPropertyPathStr.substring(
+ sourcePathStr.length()));
+ }
+
+ /** Get information of the path.
+ *
+ * @param path path for which the path information is to be fetched
+ * @param tracingContext object of tracingContext used for the tracing of the
+ * server calls.
+ *
+ * @return object containing the path information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private PathInformation getPathInformation(Path path,
+ TracingContext tracingContext)
+ throws AzureBlobFileSystemException {
+ try {
+ AbfsRestOperation op = getAbfsClient().getPathStatus(path.toString(),
+ tracingContext, null, true);
+
+ return new PathInformation(true,
+ getAbfsClient().checkIsDir(op.getResult()),
+ extractEtagHeader(op.getResult()),
+ op.getResult() instanceof AbfsHttpOperation.AbfsHttpOperationWithFixedResultForGetFileStatus);
+ } catch (AzureBlobFileSystemException e) {
Review Comment:
better to catch AbfsRestOperationException itself ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1918101261
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##########
@@ -359,5 +359,27 @@ public static String accountProperty(String property, String account) {
public static final String FS_AZURE_APACHE_HTTP_CLIENT_MAX_CACHE_CONNECTION_SIZE = "fs.azure.apache.http.client.max.cache.connection.size";
/**Maximum idle time for a ApacheHttpClient-connection: {@value}*/
public static final String FS_AZURE_APACHE_HTTP_CLIENT_IDLE_CONNECTION_TTL = "fs.azure.apache.http.client.idle.connection.ttl";
+
+ /**
+ * Blob copy API is an async API, this configuration defines polling duration
+ * for checking copy status {@value}
+ */
+ public static final String FS_AZURE_BLOB_COPY_PROGRESS_WAIT_MILLIS = "fs.azure.blob.copy.progress.wait.millis";
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1919505737
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
+ getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
+ == BlobCopyProgress.PENDING) {
+ try {
+ Thread.sleep(pollWait);
+ } catch (InterruptedException ignored) {
+
+ }
+ }
+ }
+
+ /** Gets the source path of the copy operation.
+ *
+ * @param dstPathStatus server response for the GetBlobProperties API on the
+ * destination path.
+ *
+ * @return source path of the copy operation
+ */
+ private String getDstSource(final AbfsRestOperation dstPathStatus) {
+ try {
+ String responseHeader = dstPathStatus.getResult()
+ .getResponseHeader(X_MS_COPY_SOURCE);
+ if (responseHeader == null) {
+ return null;
+ }
+ return new URL(responseHeader).toURI().getPath();
+ } catch (URISyntaxException | MalformedURLException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Verifies if the blob copy is success or a failure or still in progress.
+ *
+ * @param dstPath path of the destination for the copying
+ * @param tracingContext object of tracingContext used for the tracing of the
+ * server calls.
+ * @param copyId id returned by server on the copy server-call. This id gets
+ * attached to blob and is returned by GetBlobProperties API on the destination.
+ *
+ * @return BlobCopyProgress indicating the status of the copy operation
+ *
+ * @throws AzureBlobFileSystemException exception returned in making server call
+ * for GetBlobProperties on the path. It can be thrown if the copyStatus is failure
+ * or is aborted.
+ */
+ @VisibleForTesting
+ public BlobCopyProgress handleCopyInProgress(final Path dstPath,
+ final TracingContext tracingContext,
+ final String copyId) throws AzureBlobFileSystemException {
+ AbfsRestOperation op = getAbfsClient().getPathStatus(dstPath.toUri().getPath(),
+ tracingContext, null, false);
+
+ if (op.getResult() != null && copyId.equals(
+ op.getResult().getResponseHeader(X_MS_COPY_ID))) {
+ final String copyStatus = op.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(copyStatus)) {
+ return BlobCopyProgress.SUCCESS;
+ }
+ if (COPY_STATUS_FAILED.equalsIgnoreCase(copyStatus)) {
+ throw new AbfsRestOperationException(
+ COPY_BLOB_FAILED.getStatusCode(), COPY_BLOB_FAILED.getErrorCode(),
+ String.format("copy to path %s failed due to: %s",
+ dstPath.toUri().getPath(),
+ op.getResult().getResponseHeader(X_MS_COPY_STATUS_DESCRIPTION)),
+ new Exception(COPY_BLOB_FAILED.getErrorCode()));
+ }
+ if (COPY_STATUS_ABORTED.equalsIgnoreCase(copyStatus)) {
+ throw new AbfsRestOperationException(
+ COPY_BLOB_ABORTED.getStatusCode(), COPY_BLOB_ABORTED.getErrorCode(),
+ String.format("copy to path %s aborted", dstPath.toUri().getPath()),
+ new Exception(COPY_BLOB_ABORTED.getErrorCode()));
+ }
+ }
+ return BlobCopyProgress.PENDING;
+ }
+
+ /**
+ * Translates the destination path for a blob part of a source directory getting
+ * renamed.
+ *
+ * @param destinationDir destination directory for the rename operation
+ * @param blobPath path of blob inside sourceDir being renamed.
+ * @param sourceDir source directory for the rename operation
+ *
+ * @return translated path for the blob
+ */
+ private Path createDestinationPathForBlobPartOfRenameSrcDir(final Path destinationDir,
+ final Path blobPath, final Path sourceDir) {
+ String destinationPathStr = destinationDir.toUri().getPath();
+ String sourcePathStr = sourceDir.toUri().getPath();
+ String srcBlobPropertyPathStr = blobPath.toUri().getPath();
+ if (sourcePathStr.equals(srcBlobPropertyPathStr)) {
+ return destinationDir;
+ }
+ return new Path(
+ destinationPathStr + ROOT_PATH + srcBlobPropertyPathStr.substring(
+ sourcePathStr.length()));
+ }
+
+ /** Get information of the path.
+ *
+ * @param path path for which the path information is to be fetched
+ * @param tracingContext object of tracingContext used for the tracing of the
+ * server calls.
+ *
+ * @return object containing the path information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private PathInformation getPathInformation(Path path,
+ TracingContext tracingContext)
+ throws AzureBlobFileSystemException {
+ try {
+ AbfsRestOperation op = getAbfsClient().getPathStatus(path.toString(),
+ tracingContext, null, true);
+
+ return new PathInformation(true,
+ getAbfsClient().checkIsDir(op.getResult()),
+ extractEtagHeader(op.getResult()),
+ op.getResult() instanceof AbfsHttpOperation.AbfsHttpOperationWithFixedResultForGetFileStatus);
+ } catch (AzureBlobFileSystemException e) {
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 merged PR #7265:
URL: https://github.com/apache/hadoop/pull/7265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931701257
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931684114
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -192,10 +239,1304 @@ public void testRenameWithNoDestinationParentDir() throws Exception {
// Verify that metadata was in an incomplete state after the rename
// failure, and we retired the rename once more.
IOStatistics ioStatistics = fs.getIOStatistics();
+ AbfsClient client = fs.getAbfsStore().getClient();
IOStatisticAssertions.assertThatStatisticCounter(ioStatistics,
RENAME_PATH_ATTEMPTS.getStatName())
- .describedAs("There should be 2 rename attempts if metadata "
- + "incomplete state failure is hit")
- .isEqualTo(2);
+ .describedAs("For Dfs endpoint: There should be 2 rename "
+ + "attempts if metadata incomplete state failure is hit."
+ + "For Blob endpoint: There would be only one rename attempt which "
+ + "would have a failed precheck.")
+ .isEqualTo(client instanceof AbfsDfsClient ? 2 : 1);
+ }
+
+ @Test
+ public void testRenameToRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ fs.mkdirs(new Path("/src1/src2"));
+ assertTrue(fs.rename(new Path("/src1/src2"), new Path("/")));
+ assertTrue(fs.exists(new Path("/src2")));
+ }
+
+ @Test
+ public void testRenameNotFoundBlobToEmptyRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assertFalse(fs.rename(new Path("/file"), new Path("/")));
+ }
+
+ private void assumeNonHnsAccountBlobEndpoint(final AzureBlobFileSystem fs) {
+ assertTrue(fs.getAbfsStore().getClient() instanceof AbfsBlobClient);
+ }
+
+ @Test(expected = IOException.class)
+ public void testRenameBlobToDstWithColonInPath() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.create(new Path("/src"));
+ fs.rename(new Path("/src"), new Path("/dst:file"));
+ }
+
+ @Test
+ public void testRenameBlobInSameDirectoryWithNoMarker() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) fs.getAbfsStore().getClient();
+ fs.create(new Path("/srcDir/dir/file"));
+ client.deleteBlobPath(new Path("/srcDir/dir"), null,
+ getTestTracingContext(fs, true));
+ assertTrue(fs.rename(new Path("/srcDir/dir"), new Path("/srcDir")));
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWhereDirectoryAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.mkdirs(new Path("testDir2/test4/test3"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3")));
+ if (getIsNamespaceEnabled(fs)
+ || fs.getAbfsClient() instanceof AbfsBlobClient) {
+ assertFalse(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ } else {
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ }
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWherePartAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.create(new Path("testDir2/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("testDir2/test4/"));
+ fs.create(new Path("testDir2/test4/file1"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ Assert.assertTrue(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3")));
+ assertFalse(fs.exists(new Path("testDir2/test4/file")));
+ assertTrue(fs.exists(new Path("testDir2/test4/file1")));
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file1")));
+ assertTrue(fs.exists(new Path("testDir2/test4/file1")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ }
+
+ /**
+ * Test that after completing rename for a directory which is enabled for
+ * AtomicRename, the RenamePending JSON file is deleted.
+ */
+ @Test
+ public void testRenamePendingJsonIsRemovedPostSuccessfulRename()
+ throws Exception {
+ final AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) addSpyHooksOnClient(fs);
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.setWorkingDirectory(new Path("/"));
+ fs.mkdirs(new Path("hbase/test1/test2/test3"));
+ fs.create(new Path("hbase/test1/test2/test3/file"));
+ fs.create(new Path("hbase/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("hbase/test4/"));
+ fs.create(new Path("hbase/test4/file1"));
+ final Integer[] correctDeletePathCount = new Integer[1];
+ correctDeletePathCount[0] = 0;
+ Mockito.doAnswer(answer -> {
+ final String correctDeletePath = "/hbase/test1/test2/test3" + SUFFIX;
+ if (correctDeletePath.equals(
+ ((Path) answer.getArgument(0)).toUri().getPath())) {
+ correctDeletePathCount[0] = 1;
+ }
+ return null;
+ })
+ .when(client)
+ .deleteBlobPath(Mockito.any(Path.class), Mockito.nullable(String.class),
+ Mockito.any(TracingContext.class));
+ assertTrue(fs.rename(new Path("hbase/test1/test2/test3"),
+ new Path("hbase/test4")));
+ assertEquals("RenamePendingJson should be deleted",
+ 1,
+ (int) correctDeletePathCount[0]);
+ }
+
+ private AbfsClient addSpyHooksOnClient(final AzureBlobFileSystem fs) {
+ AzureBlobFileSystemStore store = Mockito.spy(fs.getAbfsStore());
+ Mockito.doReturn(store).when(fs).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+ return client;
+ }
+
+ /**
+ * Test for a directory in /hbase directory. To simulate the crash of process,
+ * test will throw an exception with 403 on a copy of one of the blob.<br>
+ * ListStatus API will be called on the directory. Expectation is that the ListStatus
+ * API of {@link AzureBlobFileSystem} should recover the paused rename.
+ */
+ @Test
+ public void testHBaseHandlingForFailedRenameWithListRecovery()
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931690888
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java:
##########
@@ -227,6 +236,13 @@ public void testDeleteIdempotency() throws Exception {
public void testDeleteIdempotencyTriggerHttp404() throws Exception {
final AzureBlobFileSystem fs = getFileSystem();
+ /*
+ * Delete call for a folder on DFS endpoint is one-server call and the
+ * orchestration of delete of paths inside the directory. For Blob
+ * endpoint, the orchestration would be done by the client. The idempotency
+ * issue would not happen for blob endpoint.
+ */
+ assertTrue(fs.getAbfsClient() instanceof AbfsDfsClient);
Review Comment:
This method is not yet present in trunk. I guess this method is coming with Ingress Handler changes. Will update this later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933468312
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsPaginatedDelete.java:
##########
@@ -142,6 +142,7 @@ private void setTestUserConf(Configuration conf, String key, String value) {
*/
@Test
public void testRecursiveDeleteWithPagination() throws Exception {
+ assertTrue(getFileSystem().getAbfsStore().getClient() instanceof AbfsDfsClient);
Review Comment:
Should it be assume here and other places in file??
Also instead of checking on instance of you can use methods defined in base class for getting current servce type and use them
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java:
##########
@@ -452,7 +452,6 @@ public static AbfsClient getMockAbfsClient(AbfsClient baseAbfsClientInstance,
(currentAuthType == AuthType.SharedKey)
|| (currentAuthType == AuthType.OAuth));
- // TODO : [FnsOverBlob][HADOOP-19234] Update to work with Blob Endpoint as well when Fns Over Blob is ready.
Review Comment:
Why we are removing this, we still not to update test to work with Blob Client
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRenameUnicode.java:
##########
@@ -26,13 +26,19 @@
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
Review Comment:
Import ordering fix
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -18,36 +18,83 @@
package org.apache.hadoop.fs.azurebfs;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.AccessDeniedException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
-import org.junit.Assert;
+import org.apache.hadoop.fs.FileSystem;
Review Comment:
Import ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933352626
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -351,7 +395,7 @@ public void testPosixRenameDirectoryWherePartAlreadyThereOnDestination()
fs.create(new Path("testDir2/test4/file1"));
assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
- Assert.assertTrue(fs.rename(new Path("testDir2/test1/test2/test3"),
+ assertTrue(fs.rename(new Path("testDir2/test1/test2/test3"),
Review Comment:
same assertions.assertThat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933456710
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1063,11 +1085,6 @@ public boolean rename(final Path source,
long countAggregate = 0;
boolean shouldContinue;
- if (isAtomicRenameKey(source.getName())) {
Review Comment:
With the atomic rename feature now implemented for blob, we are removing this check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936788726
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCreate.java:
##########
@@ -171,50 +169,130 @@ public void testCreateNonRecursive2() throws Exception {
assertIsFile(fs, testFile);
}
+ private void assumeNonHnsAccountBlobEndpoint(final AzureBlobFileSystem fs) {
+ Assume.assumeTrue(fs.getAbfsStore().getClient() instanceof AbfsBlobClient);
+ }
+
+ /**
+ * Test createNonRecursive when parent exist.
+ *
+ * @throws Exception in case of failure
+ */
+ @Test
+ public void testCreateNonRecursiveWhenParentExist() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path createDirectoryPath = new Path("hbase/A");
+ fs.mkdirs(createDirectoryPath);
+ fs.createNonRecursive(new Path(createDirectoryPath, "B"), FsPermission
+ .getDefault(), false, 1024, (short) 1, 1024, null);
+ Assertions.assertThat(fs.exists(new Path(createDirectoryPath, "B"))).isTrue();
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2627464063
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 14 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 18s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 18s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 31s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 115m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/26/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 497431a9a472 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a4e97cb974152aa2960c8c68e6996d460ddfebaa |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/26/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/26/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2627487350
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 14 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 43s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 22 unchanged - 4 fixed = 22 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 25s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 130m 18s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/25/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux e1d954aeec0b 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / cc85e802a7da708c9ae3be11a960d5d7acc6371b |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/25/testReport/ |
| Max. process+thread count | 528 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/25/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: [ABFS] Support Rename and Delete operation over FNS-Blob endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2567470188
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 44s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 3s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 13 new + 7 unchanged - 3 fixed = 20 total (was 10) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 28s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 25s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| -1 :x: | spotbugs | 1m 9s | [/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/1/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 40m 3s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 39s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 135m 25s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Nullcheck of tracingContext at line 837 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsBlobClient.getPathStatus(String, boolean, TracingContext, ContextEncryptionAdapter) At AbfsBlobClient.java:837 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsBlobClient.getPathStatus(String, boolean, TracingContext, ContextEncryptionAdapter) At AbfsBlobClient.java:[line 831] |
| | Random object created and used only once in org.apache.hadoop.fs.azurebfs.services.RenameAtomicity.createRenamePendingJson(Path, byte[]) At RenameAtomicity.java:only once in org.apache.hadoop.fs.azurebfs.services.RenameAtomicity.createRenamePendingJson(Path, byte[]) At RenameAtomicity.java:[line 181] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19381 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 390fc4d30d87 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 74d17604fab47f3f585447410cbb22de9ce7c757 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/1/testReport/ |
| Max. process+thread count | 542 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: [ABFS] Support Rename and Delete operation over FNS-Blob endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2568856958
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 5s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 7 unchanged - 3 fixed = 8 total (was 10) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 29s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 10 new + 11 unchanged - 0 fixed = 21 total (was 11) |
| -1 :x: | javadoc | 0m 26s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 10 new + 11 unchanged - 0 fixed = 21 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 0s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 38s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 135m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a0f8592c7fd2 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bcf554436d7202442eb81e33901166e608ec8f3a |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/2/testReport/ |
| Max. process+thread count | 607 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2618800072
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 17m 42s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 23s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 41s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/10/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 3 new + 13 unchanged - 3 fixed = 16 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 28s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/10/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 3 new + 10 unchanged - 1 fixed = 13 total (was 11) |
| -1 :x: | javadoc | 0m 24s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/10/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 3 new + 10 unchanged - 1 fixed = 13 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 34s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 146m 41s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 10960cb2dbd9 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d2b422d397b9ff4091cd71bcc1770418e0fe8189 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/10/testReport/ |
| Max. process+thread count | 586 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2619135190
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 50s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 13 unchanged - 3 fixed = 13 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 25s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 130m 12s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 19cdd1c3b260 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9d9fb96a176715fb4ad42d2a23cafa1c29446d85 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/11/testReport/ |
| Max. process+thread count | 525 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1910280163
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -338,6 +353,7 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
requestHeaders);
op.execute(tracingContext);
+ fixAtomicEntriesInListResults(op, tracingContext);
Review Comment:
explain the need for the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936666740
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -909,23 +1025,46 @@ public AbfsRestOperation read(final String path,
}
/**
- * Orchestration for delete operation to be implemented.
+ * Get Rest Operation for API <a href = https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete></a>.
Review Comment:
Not able to see this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936799344
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/SASTokenProvider.java:
##########
@@ -49,6 +49,9 @@ public interface SASTokenProvider {
String SET_PERMISSION_OPERATION = "set-permission";
String SET_PROPERTIES_OPERATION = "set-properties";
String WRITE_OPERATION = "write";
+ String COPY_BLOB_DESTINATION = "copy-blob-dst";
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936904569
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsPaginatedDelete.java:
##########
@@ -142,6 +142,7 @@ private void setTestUserConf(Configuration conf, String key, String value) {
*/
@Test
public void testRecursiveDeleteWithPagination() throws Exception {
+ assertTrue(getFileSystem().getAbfsStore().getClient() instanceof AbfsDfsClient);
Review Comment:
Here we are checking for AbfsDfsClient and there is no common method to check on DFS client. This is part of Ingress PR and it is already merged to trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936995569
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,634 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.TimeoutException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COLON;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = getPathInformation(src, tracingContext);
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(
+ src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * create renameJson file.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as rename on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(),
+ false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(COLON);
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path, getDstPathForBlob(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus != null && dstPathStatus.getResult() != null
+ && (srcCopyPath.equals(getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ final long maxWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressMaxWaitMillis();
+ long startTime = System.currentTimeMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
+ == BlobCopyProgress.PENDING) {
+ if (System.currentTimeMillis() - startTime > maxWait) {
+ throw new TimeoutException(
+ String.format("Blob copy progress wait time exceeded "
Review Comment:
nit : LOG.debug("Loaded map stats - File: {}, Loaded: {}, Error: {} ", fileLocation, cache.size(), errorRecord); -> This is the format we use across the repo {optional}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933627680
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java:
##########
@@ -452,7 +452,6 @@ public static AbfsClient getMockAbfsClient(AbfsClient baseAbfsClientInstance,
(currentAuthType == AuthType.SharedKey)
|| (currentAuthType == AuthType.OAuth));
- // TODO : [FnsOverBlob][HADOOP-19234] Update to work with Blob Endpoint as well when Fns Over Blob is ready.
Review Comment:
Reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2624547342
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 49s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 21s | [/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 23s | [/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 0m 23s | [/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 0m 21s | [/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| -1 :x: | javac | 0m 21s | [/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 13 unchanged - 4 fixed = 14 total (was 17) |
| -1 :x: | mvnsite | 0m 22s | [/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | javadoc | 0m 21s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 24s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 4 new + 10 unchanged - 1 fixed = 14 total (was 11) |
| -1 :x: | spotbugs | 0m 20s | [/patch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/patch-spotbugs-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +1 :green_heart: | shadedclient | 40m 59s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 25s | [/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 130m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 2b75e60bd980 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fd8957fce72aa614678e203517a7f0f5cebc9431 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/testReport/ |
| Max. process+thread count | 617 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2631550062
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 21 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 38s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 25s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 38s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 131m 29s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/33/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 691b86429183 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d2c8baf037c198d48e6c8749676dbbb5080d067d |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/33/testReport/ |
| Max. process+thread count | 620 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/33/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916387771
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
+ getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
Review Comment:
check copyId != null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916391781
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
+ getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
Review Comment:
If handleCopyInProgress keeps returning PENDING, the code might enter an infinite loop of waiting. We should introduce maximum wait time and if exceeded fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917834953
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/enums/BlobCopyProgress.java:
##########
@@ -0,0 +1,26 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.enums;
+
+public enum BlobCopyProgress {
Review Comment:
Added java Doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1918136851
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HttpHeaderConfigurations.java:
##########
@@ -104,5 +104,9 @@ public final class HttpHeaderConfigurations {
*/
public static final String X_MS_BLOB_CONTENT_MD5 = "x-ms-blob-content-md5";
+ public static final String X_MS_COPY_ID = "x-ms-copy-id";
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2578002647
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 57s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 3s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/5/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 26 new + 13 unchanged - 3 fixed = 39 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| -1 :x: | javadoc | 0m 27s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/5/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 2 new + 10 unchanged - 1 fixed = 12 total (was 11) |
| -1 :x: | javadoc | 0m 25s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/5/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 2 new + 10 unchanged - 1 fixed = 12 total (was 11) |
| -1 :x: | spotbugs | 1m 7s | [/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/5/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 32m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 27s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 116m 59s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Dead store to blockIdList in org.apache.hadoop.fs.azurebfs.services.RenameAtomicity.createRenamePendingJson(Path, byte[]) At RenameAtomicity.java:org.apache.hadoop.fs.azurebfs.services.RenameAtomicity.createRenamePendingJson(Path, byte[]) At RenameAtomicity.java:[line 231] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux eddd7e0f4803 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5a29c44579ec6e866904aa5c4bebc33ce891f842 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/5/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1904918517
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java:
##########
@@ -201,4 +238,25 @@ public int getAcquireRetryCount() {
public TracingContext getTracingContext() {
return tracingContext;
}
+
Review Comment:
Java docs for all public methods and classes
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/SASTokenProvider.java:
##########
@@ -49,6 +49,9 @@ public interface SASTokenProvider {
String SET_PERMISSION_OPERATION = "set-permission";
String SET_PROPERTIES_OPERATION = "set-properties";
String WRITE_OPERATION = "write";
+ String COPY_BLOB_DESTINATION = "copy-blob-dst";
Review Comment:
Need to discuss this change once, we do not support UDS for FNS Blob
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/enums/BlobCopyProgress.java:
##########
@@ -0,0 +1,26 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.enums;
+
+public enum BlobCopyProgress {
Review Comment:
Javadoc for class and enums
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/AzureServiceErrorCode.java:
##########
@@ -54,10 +56,13 @@ public enum AzureServiceErrorCode {
OTHER_SERVER_THROTTLING("ServerBusy", HttpURLConnection.HTTP_UNAVAILABLE,
"The server is currently unable to receive requests. Please retry your request."),
INVALID_QUERY_PARAMETER_VALUE("InvalidQueryParameterValue", HttpURLConnection.HTTP_BAD_REQUEST, null),
+ INVALID_RENAME_DESTINATION("InvalidRenameDestinationPath", HttpURLConnection.HTTP_BAD_REQUEST, null),
AUTHORIZATION_PERMISSION_MISS_MATCH("AuthorizationPermissionMismatch", HttpURLConnection.HTTP_FORBIDDEN, null),
ACCOUNT_REQUIRES_HTTPS("AccountRequiresHttps", HttpURLConnection.HTTP_BAD_REQUEST, null),
MD5_MISMATCH("Md5Mismatch", HttpURLConnection.HTTP_BAD_REQUEST,
"The MD5 value specified in the request did not match with the MD5 value calculated by the server."),
+ COPY_BLOB_FAILED("COPY_BLOB_FAILED", HttpURLConnection.HTTP_INTERNAL_ERROR, null),
Review Comment:
Error Codes should be in camelcase as others
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -909,23 +959,32 @@ public AbfsRestOperation read(final String path,
}
/**
- * Orchestration for delete operation to be implemented.
+ * Get Rest Operation for API <a href = https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete></a>.
+ * Delete the file or directory at specified path.
* @param path to be deleted.
* @param recursive if the path is a directory, delete recursively.
* @param continuation to specify continuation token.
- * @param tracingContext for tracing the server calls.
- * @param isNamespaceEnabled specify if the namespace is enabled.
+ * @param tracingContext TracingContext instance to track identifiers
+ * @param isNamespaceEnabled whether namespace enabled account or not
* @return executed rest operation containing response from server.
* @throws AzureBlobFileSystemException if rest operation fails.
*/
@Override
public AbfsRestOperation deletePath(final String path,
- final boolean recursive,
- final String continuation,
- TracingContext tracingContext,
- final boolean isNamespaceEnabled) throws AzureBlobFileSystemException {
- // TODO: [FnsOverBlob][HADOOP-19233] To be implemented as part of rename-delete over blob endpoint work.
- throw new NotImplementedException("Delete operation on Blob endpoint will be implemented in future.");
+ final boolean recursive,
+ final String continuation,
+ final TracingContext tracingContext,
+ final boolean isNamespaceEnabled) throws AzureBlobFileSystemException {
+ getBlobDeleteHandler(path, recursive, tracingContext).execute();
+ return null;
Review Comment:
Why are we returning null here?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -499,21 +526,36 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws AzureBlobFileSystemException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
- final String destination,
- final String continuation,
- final TracingContext tracingContext,
- final String sourceEtag,
- final boolean isMetadataIncompleteState,
- final boolean isNamespaceEnabled) throws IOException {
- /**
- * TODO: [FnsOverBlob] To be implemented as part of rename-delete over blob endpoint work. <a href="https://issues.apache.org/jira/browse/HADOOP-19233">HADOOP-19233</a>.
- */
- throw new NotImplementedException("Rename operation on Blob endpoint yet to be implemented.");
+ final String destination,
+ final String continuation,
+ final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
+ throws IOException {
+ BlobRenameHandler blobRenameHandler = getBlobRenameHandler(source,
+ destination, sourceEtag, isAtomicRenameKey(source), tracingContext
+ );
+ incrementAbfsRenamePath();
+ return blobRenameHandler.execute();
Review Comment:
This might need rechecking. We do not want to return op as null.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/PathInformation.java:
##########
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
Review Comment:
Javadocs missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916382723
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
Review Comment:
check for copyPathOp != null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916382723
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
Review Comment:
check for copyPathOp != null && copyPathOp.getResult() != null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916372687
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
Review Comment:
Method name can be shortened
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1919502056
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
+//import java.util.Collections;
+//import java.util.List;
+import java.util.Random;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsDriverException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.contracts.services.BlobAppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static java.net.HttpURLConnection.HTTP_PRECON_FAILED;
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.BLOCK_ID_LENGTH;
+//import static org.apache.hadoop.fs.azurebfs.services.AzureIngressHandler.generateBlockListXml;
+
+/**
+ * For a directory enabled for atomic-rename, before rename starts, a file with
+ * -RenamePending.json suffix is created. In this file, the states required for the
+ * rename operation are given. This file is created by {@link #preRename()} method.
+ * This is important in case the JVM process crashes during rename, the atomicity
+ * will be maintained, when the job calls {@link AzureBlobFileSystem#listStatus(Path)}
+ * or {@link AzureBlobFileSystem#getFileStatus(Path)}. On these API calls to filesystem,
+ * it will be checked if there is any RenamePending JSON file. If yes, the crashed rename
+ * operation would be resumed as per the file.
+ */
+public class RenameAtomicity {
+
+ private final TracingContext tracingContext;
+
+ private Path src, dst;
+
+ private String srcEtag;
+
+ private final AbfsBlobClient abfsClient;
+
+ private final Path renameJsonPath;
+
+ public static final String SUFFIX = "-RenamePending.json";
+
+ private int preRenameRetryCount = 0;
+
+ private int renamePendingJsonLen;
+
+ private final AbfsLease sourcePathLease;
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private static final Random RANDOM = new Random();
+
+ /**
+ * Performs pre-rename operations. Creates a file with -RenamePending.json
+ * suffix in the source parent directory. This file contains the states
+ * required for the rename operation.
+ *
+ * @param src Source path
+ * @param dst Destination path
+ * @param renameJsonPath Path of the JSON file to be created
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ */
+ public RenameAtomicity(final Path src, final Path dst,
+ final Path renameJsonPath,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient) {
+ this.src = src;
+ this.dst = dst;
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.sourcePathLease = null;
+ }
+
+ /**
+ * Resumes the rename operation from the JSON file.
+ *
+ * @param renameJsonPath Path of the JSON file
+ * @param renamePendingJsonFileLen Length of the JSON file
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ * @param sourceLease Lease of the source directory
+ */
+ public RenameAtomicity(final Path renameJsonPath,
+ final int renamePendingJsonFileLen,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient,
+ final AbfsLease sourceLease) {
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.renamePendingJsonLen = renamePendingJsonFileLen;
+ this.sourcePathLease = sourceLease;
+ }
+
+ /**
+ * Redo the rename operation from the JSON file.
+ *
+ * @throws AzureBlobFileSystemException If the redo operation fails.
+ */
+ public void redo() throws AzureBlobFileSystemException {
+ byte[] buffer = readRenamePendingJson(renameJsonPath, renamePendingJsonLen);
+ String contents = new String(buffer, Charset.defaultCharset());
+ try {
+ final RenamePendingJsonFormat renamePendingJsonFormatObj;
+ try {
+ renamePendingJsonFormatObj = objectMapper.readValue(contents,
+ RenamePendingJsonFormat.class);
+ } catch (JsonProcessingException e) {
+ return;
Review Comment:
Yes, it is expected. In case of Json Processing Error, we are deleting the json file and returning from it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917834953
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/enums/BlobCopyProgress.java:
##########
@@ -0,0 +1,26 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.enums;
+
+public enum BlobCopyProgress {
Review Comment:
Java Doc added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917794969
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/AzureServiceErrorCode.java:
##########
@@ -54,10 +56,13 @@ public enum AzureServiceErrorCode {
OTHER_SERVER_THROTTLING("ServerBusy", HttpURLConnection.HTTP_UNAVAILABLE,
"The server is currently unable to receive requests. Please retry your request."),
INVALID_QUERY_PARAMETER_VALUE("InvalidQueryParameterValue", HttpURLConnection.HTTP_BAD_REQUEST, null),
+ INVALID_RENAME_DESTINATION("InvalidRenameDestinationPath", HttpURLConnection.HTTP_BAD_REQUEST, null),
AUTHORIZATION_PERMISSION_MISS_MATCH("AuthorizationPermissionMismatch", HttpURLConnection.HTTP_FORBIDDEN, null),
ACCOUNT_REQUIRES_HTTPS("AccountRequiresHttps", HttpURLConnection.HTTP_BAD_REQUEST, null),
MD5_MISMATCH("Md5Mismatch", HttpURLConnection.HTTP_BAD_REQUEST,
"The MD5 value specified in the request did not match with the MD5 value calculated by the server."),
+ COPY_BLOB_FAILED("COPY_BLOB_FAILED", HttpURLConnection.HTTP_INTERNAL_ERROR, null),
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2578651373
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 17m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 48s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/6/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 6 new + 13 unchanged - 3 fixed = 19 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| -1 :x: | spotbugs | 1m 10s | [/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/6/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 40m 8s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 39s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 152m 56s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Dead store to blockIdList in org.apache.hadoop.fs.azurebfs.services.RenameAtomicity.createRenamePendingJson(Path, byte[]) At RenameAtomicity.java:org.apache.hadoop.fs.azurebfs.services.RenameAtomicity.createRenamePendingJson(Path, byte[]) At RenameAtomicity.java:[line 231] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d7976fb5527c 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 83fc86a8b0978e8620b74f81dc59bfa62127b586 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/6/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931744572
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
Review Comment:
As discussed offline, this is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933336923
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -174,7 +151,7 @@ public boolean execute() throws AzureBlobFileSystemException {
if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
/*
* if it is not a resume of a previous failed atomic rename operation,
- * perform the pre-rename operation.
+ * Create the rename JSON.
Review Comment:
nit: create (C small)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937122362
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsPermission.java:
##########
@@ -68,7 +69,8 @@ public boolean equals(Object obj) {
* @return a permission object for the provided string representation
*/
public static AbfsPermission valueOf(final String abfsSymbolicPermission) {
- if (abfsSymbolicPermission == null) {
+ if (abfsSymbolicPermission == null
Review Comment:
Taken
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCreate.java:
##########
@@ -166,6 +169,147 @@ public void testCreateNonRecursive2() throws Exception {
assertIsFile(fs, testFile);
}
+ /**
+ * Test createNonRecursive when parent exist.
+ *
+ * @throws Exception in case of failure
+ */
+ @Test
+ public void testCreateNonRecursiveWhenParentExist() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeBlobServiceType();
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path createDirectoryPath = new Path("hbase/A");
+ fs.mkdirs(createDirectoryPath);
+ fs.createNonRecursive(new Path(createDirectoryPath, "B"), FsPermission
+ .getDefault(), false, 1024,
+ (short) 1, 1024, null);
+ Assertions.assertThat(fs.exists(new Path(createDirectoryPath, "B")))
+ .describedAs("File should be created").isTrue();
+ fs.close();
+ }
+
+ /**
+ * Test createNonRecursive when parent does not exist.
+ *
+ * @throws Exception in case of failure
+ */
+ @Test
+ public void testCreateNonRecursiveWhenParentNotExist() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeBlobServiceType();
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path createDirectoryPath = new Path("A/");
Review Comment:
Both works.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1694,16 +1976,24 @@ private boolean isNonEmptyListing(String path,
* @return True if empty results without continuation token.
*/
private boolean isEmptyListResults(AbfsHttpOperation result) {
Review Comment:
Taken
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1144,9 +1158,9 @@ public void delete(final Path path, final boolean recursive,
boolean shouldContinue = true;
LOG.debug("delete filesystem: {} path: {} recursive: {}",
- getClient().getFileSystem(),
- path,
- String.valueOf(recursive));
+ getClient().getFileSystem(),
Review Comment:
Same as above, the formatting was not as per out XML, so I thought to correct it for delete flow.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsPermission.java:
##########
@@ -100,7 +102,8 @@ public static AbfsPermission valueOf(final String abfsSymbolicPermission) {
* extended ACL; otherwise false.
*/
public static boolean isExtendedAcl(final String abfsSymbolicPermission) {
- if (abfsSymbolicPermission == null) {
+ if (abfsSymbolicPermission == null
Review Comment:
Taken
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1155,9 +1169,10 @@ public void delete(final Path path, final boolean recursive,
do {
try (AbfsPerfInfo perfInfo = startTracking("delete", "deletePath")) {
AbfsRestOperation op = getClient().deletePath(relativePath, recursive,
- continuation, tracingContext, getIsNamespaceEnabled(tracingContext));
+ continuation, tracingContext);
perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ continuation = op.getResult()
Review Comment:
Corrected formatting only in rename and delete flow as earlier formatting was not as per what we follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937136064
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -673,7 +694,7 @@ public OutputStream createFile(final Path path,
}
final ContextEncryptionAdapter contextEncryptionAdapter;
- if (createClient.getEncryptionType() == EncryptionType.ENCRYPTION_CONTEXT) {
+ if (getClient().getEncryptionType() == EncryptionType.ENCRYPTION_CONTEXT) {
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936114001
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -609,10 +694,12 @@ public AbfsRestOperation acquireLease(final String path, final int duration,
public AbfsRestOperation renewLease(final String path, final String leaseId,
TracingContext tracingContext) throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
- requestHeaders.add(new AbfsHttpHeader(X_MS_LEASE_ACTION, RENEW_LEASE_ACTION));
+ requestHeaders.add(
+ new AbfsHttpHeader(X_MS_LEASE_ACTION, RENEW_LEASE_ACTION));
requestHeaders.add(new AbfsHttpHeader(X_MS_LEASE_ID, leaseId));
- final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
+ final AbfsUriQueryBuilder abfsUriQueryBuilder
Review Comment:
same as above: undo changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936700770
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -689,21 +789,53 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws IOException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
final String destination,
final String continuation,
final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
Review Comment:
remove isNamespaceEnabled as param and take the value from the method call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936795920
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java:
##########
@@ -227,6 +236,13 @@ public void testDeleteIdempotency() throws Exception {
public void testDeleteIdempotencyTriggerHttp404() throws Exception {
final AzureBlobFileSystem fs = getFileSystem();
+ /*
+ * Delete call for a folder on DFS endpoint is one-server call and the
+ * orchestration of delete of paths inside the directory. For Blob
+ * endpoint, the orchestration would be done by the client. The idempotency
+ * issue would not happen for blob endpoint.
+ */
+ assertTrue(fs.getAbfsClient() instanceof AbfsDfsClient);
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2624600391
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 56s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 7s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 22s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/17/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 14 unchanged - 3 fixed = 15 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 33s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 116m 0s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 7dc54bbc8996 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2166246fcdcfc4c1b71a5d4e766a35da3507697b |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/17/testReport/ |
| Max. process+thread count | 546 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/17/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936983225
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1155,9 +1169,11 @@ public void delete(final Path path, final boolean recursive,
do {
try (AbfsPerfInfo perfInfo = startTracking("delete", "deletePath")) {
AbfsRestOperation op = getClient().deletePath(relativePath, recursive,
- continuation, tracingContext, getIsNamespaceEnabled(tracingContext));
+ continuation, tracingContext,
Review Comment:
isnamespaceenabled should be removed from here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2628951790
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 14 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 20s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 9s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 132m 38s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/28/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 4f80936e398e 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f00f5c97ce8ae2292c3516e4cd4671f989af26a5 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/28/testReport/ |
| Max. process+thread count | 530 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/28/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937112476
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##########
@@ -21,25 +21,24 @@
import java.io.IOException;
import java.lang.reflect.Field;
-import org.apache.hadoop.classification.VisibleForTesting;
-import org.apache.hadoop.fs.azurebfs.constants.AbfsServiceType;
-import org.apache.hadoop.fs.azurebfs.services.FixedSASTokenProvider;
-import org.apache.hadoop.fs.azurebfs.constants.HttpOperationType;
-import org.apache.hadoop.fs.azurebfs.utils.MetricFormat;
-import org.apache.hadoop.util.Preconditions;
+import org.slf4j.Logger;
Review Comment:
Thank you for fixing import ordering in this class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933351513
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java:
##########
@@ -314,4 +347,278 @@ public void deleteBlobDirParallelThreadToDeleteOnDifferentTracingContext()
fs.delete(new Path("/testDir"), true);
fs.close();
}
+
+ private void assumeBlobClient() throws IOException {
Review Comment:
It is available in AbstractAbfsIntegrationTest class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931687188
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -192,10 +239,1304 @@ public void testRenameWithNoDestinationParentDir() throws Exception {
// Verify that metadata was in an incomplete state after the rename
// failure, and we retired the rename once more.
IOStatistics ioStatistics = fs.getIOStatistics();
+ AbfsClient client = fs.getAbfsStore().getClient();
IOStatisticAssertions.assertThatStatisticCounter(ioStatistics,
RENAME_PATH_ATTEMPTS.getStatName())
- .describedAs("There should be 2 rename attempts if metadata "
- + "incomplete state failure is hit")
- .isEqualTo(2);
+ .describedAs("For Dfs endpoint: There should be 2 rename "
+ + "attempts if metadata incomplete state failure is hit."
+ + "For Blob endpoint: There would be only one rename attempt which "
+ + "would have a failed precheck.")
+ .isEqualTo(client instanceof AbfsDfsClient ? 2 : 1);
+ }
+
+ @Test
+ public void testRenameToRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ fs.mkdirs(new Path("/src1/src2"));
+ assertTrue(fs.rename(new Path("/src1/src2"), new Path("/")));
+ assertTrue(fs.exists(new Path("/src2")));
+ }
+
+ @Test
+ public void testRenameNotFoundBlobToEmptyRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assertFalse(fs.rename(new Path("/file"), new Path("/")));
+ }
+
+ private void assumeNonHnsAccountBlobEndpoint(final AzureBlobFileSystem fs) {
+ assertTrue(fs.getAbfsStore().getClient() instanceof AbfsBlobClient);
+ }
+
+ @Test(expected = IOException.class)
+ public void testRenameBlobToDstWithColonInPath() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.create(new Path("/src"));
+ fs.rename(new Path("/src"), new Path("/dst:file"));
+ }
+
+ @Test
+ public void testRenameBlobInSameDirectoryWithNoMarker() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) fs.getAbfsStore().getClient();
+ fs.create(new Path("/srcDir/dir/file"));
+ client.deleteBlobPath(new Path("/srcDir/dir"), null,
+ getTestTracingContext(fs, true));
+ assertTrue(fs.rename(new Path("/srcDir/dir"), new Path("/srcDir")));
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWhereDirectoryAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.mkdirs(new Path("testDir2/test4/test3"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3")));
+ if (getIsNamespaceEnabled(fs)
+ || fs.getAbfsClient() instanceof AbfsBlobClient) {
+ assertFalse(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ } else {
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ }
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWherePartAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.create(new Path("testDir2/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("testDir2/test4/"));
+ fs.create(new Path("testDir2/test4/file1"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ Assert.assertTrue(fs.rename(new Path("testDir2/test1/test2/test3"),
Review Comment:
Used assertTrue from Assertion class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937021825
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsRenameRetryRecovery.java:
##########
@@ -434,6 +437,9 @@ public void testExistingPathCorrectlyRejected() throws Exception {
@Test
public void testRenameRecoveryUnsupportedForFlatNamespace() throws Exception {
Assume.assumeTrue(!isNamespaceEnabled);
+ // In DFS endpoint, renamePath is O(1) API call and idempotency issue can happen.
+ // For blob endpoint, client orchestrates the rename operation.
+ Assume.assumeTrue(getFileSystem().getAbfsStore().getClient() instanceof AbfsDfsClient);
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1938969052
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ListBlobQueue.java:
##########
@@ -170,7 +170,7 @@ private List<Path> dequeue() {
return pathListForConsumption;
}
- private synchronized int size() {
+ synchronized int size() {
Review Comment:
need for this removal ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1938978707
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -535,34 +541,20 @@ public boolean rename(final Path src, final Path dst) throws IOException {
return tryGetFileStatus(qualifiedSrcPath, tracingContext) != null;
}
- FileStatus dstFileStatus = null;
+ FileStatus dstFileStatus = tryGetFileStatus(qualifiedDstPath, tracingContext);
+ Path adjustedDst = dst;
if (qualifiedSrcPath.equals(qualifiedDstPath)) {
- // rename to itself
- // - if it doesn't exist, return false
- // - if it is file, return true
- // - if it is dir, return false.
- dstFileStatus = tryGetFileStatus(qualifiedDstPath, tracingContext);
- if (dstFileStatus == null) {
+ return dstFileStatus != null && !dstFileStatus.isDirectory();
+ } else if (dstFileStatus != null) {
Review Comment:
In case dstFileStatus is null, we don't want to adjust the destination Path. Thats why this if else is needed.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -649,18 +647,16 @@ public void deleteFilesystem(TracingContext tracingContext)
*
* @throws IOException if there is an issue with the operation.
*/
- public void createNonRecursivePreCheck(final Path path,
+ public OutputStream createNonRecursive(final Path path,
+ final FileSystem.Statistics statistics, final boolean overwrite,
+ final FsPermission permission, final FsPermission umask,
TracingContext tracingContext)
throws IOException {
- if (path.getParent() != null) {
- getClient().createNonRecursivePreCheck(path.getParent(),
- tracingContext);
- } else {
- throw new AbfsRestOperationException(HTTP_CONFLICT,
- AzureServiceErrorCode.PATH_CONFLICT.getErrorCode(),
- ERR_CREATE_ON_ROOT,
- null);
- }
+ getClient().createNonRecursivePreCheck(path.getParent(),
Review Comment:
Will add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1912740827
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -909,23 +959,32 @@ public AbfsRestOperation read(final String path,
}
/**
- * Orchestration for delete operation to be implemented.
+ * Get Rest Operation for API <a href = https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete></a>.
+ * Delete the file or directory at specified path.
* @param path to be deleted.
* @param recursive if the path is a directory, delete recursively.
* @param continuation to specify continuation token.
- * @param tracingContext for tracing the server calls.
- * @param isNamespaceEnabled specify if the namespace is enabled.
+ * @param tracingContext TracingContext instance to track identifiers
+ * @param isNamespaceEnabled whether namespace enabled account or not
* @return executed rest operation containing response from server.
* @throws AzureBlobFileSystemException if rest operation fails.
*/
@Override
public AbfsRestOperation deletePath(final String path,
- final boolean recursive,
- final String continuation,
- TracingContext tracingContext,
- final boolean isNamespaceEnabled) throws AzureBlobFileSystemException {
- // TODO: [FnsOverBlob][HADOOP-19233] To be implemented as part of rename-delete over blob endpoint work.
- throw new NotImplementedException("Delete operation on Blob endpoint will be implemented in future.");
+ final boolean recursive,
+ final String continuation,
+ final TracingContext tracingContext,
+ final boolean isNamespaceEnabled) throws AzureBlobFileSystemException {
+ getBlobDeleteHandler(path, recursive, tracingContext).execute();
+ return null;
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1910277487
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##########
@@ -359,5 +359,27 @@ public static String accountProperty(String property, String account) {
public static final String FS_AZURE_APACHE_HTTP_CLIENT_MAX_CACHE_CONNECTION_SIZE = "fs.azure.apache.http.client.max.cache.connection.size";
/**Maximum idle time for a ApacheHttpClient-connection: {@value}*/
public static final String FS_AZURE_APACHE_HTTP_CLIENT_IDLE_CONNECTION_TTL = "fs.azure.apache.http.client.idle.connection.ttl";
+
+ /**
+ * Blob copy API is an async API, this configuration defines polling duration
+ * for checking copy status {@value}
+ */
+ public static final String FS_AZURE_BLOB_COPY_PROGRESS_WAIT_MILLIS = "fs.azure.blob.copy.progress.wait.millis";
Review Comment:
keep the comments formatting constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2624465496
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 29s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 51s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 21s | [/patch-mvninstall-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | compile | 0m 23s | [/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 0m 23s | [/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 0m 21s | [/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| -1 :x: | javac | 0m 21s | [/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/patch-compile-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 12 unchanged - 5 fixed = 13 total (was 17) |
| -1 :x: | mvnsite | 0m 23s | [/patch-mvnsite-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/patch-mvnsite-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| -1 :x: | javadoc | 0m 21s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 25s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 4 new + 10 unchanged - 1 fixed = 14 total (was 11) |
| -1 :x: | spotbugs | 0m 21s | [/patch-spotbugs-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/patch-spotbugs-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +1 :green_heart: | shadedclient | 40m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 26s | [/patch-unit-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt) | hadoop-azure in the patch failed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 128m 45s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 710709f77fef 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c92e00a7d47646a55715aea746a4192e255f2f73 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/testReport/ |
| Max. process+thread count | 635 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/14/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936998968
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCreate.java:
##########
@@ -166,6 +169,147 @@ public void testCreateNonRecursive2() throws Exception {
assertIsFile(fs, testFile);
}
+ /**
+ * Test createNonRecursive when parent exist.
+ *
+ * @throws Exception in case of failure
+ */
+ @Test
+ public void testCreateNonRecursiveWhenParentExist() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeBlobServiceType();
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path createDirectoryPath = new Path("hbase/A");
+ fs.mkdirs(createDirectoryPath);
+ fs.createNonRecursive(new Path(createDirectoryPath, "B"), FsPermission
+ .getDefault(), false, 1024,
+ (short) 1, 1024, null);
+ Assertions.assertThat(fs.exists(new Path(createDirectoryPath, "B")))
+ .describedAs("File should be created").isTrue();
+ fs.close();
+ }
+
+ /**
+ * Test createNonRecursive when parent does not exist.
+ *
+ * @throws Exception in case of failure
+ */
+ @Test
+ public void testCreateNonRecursiveWhenParentNotExist() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeBlobServiceType();
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path createDirectoryPath = new Path("A/");
Review Comment:
should this be /A/ or just /A ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937002186
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsRenameRetryRecovery.java:
##########
@@ -434,6 +437,9 @@ public void testExistingPathCorrectlyRejected() throws Exception {
@Test
public void testRenameRecoveryUnsupportedForFlatNamespace() throws Exception {
Assume.assumeTrue(!isNamespaceEnabled);
+ // In DFS endpoint, renamePath is O(1) API call and idempotency issue can happen.
+ // For blob endpoint, client orchestrates the rename operation.
+ Assume.assumeTrue(getFileSystem().getAbfsStore().getClient() instanceof AbfsDfsClient);
Review Comment:
Now we have getAbfsServiceType() method in trunk so can be used everywhere where we are checking for client, we can directly check for service type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937002186
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsRenameRetryRecovery.java:
##########
@@ -434,6 +437,9 @@ public void testExistingPathCorrectlyRejected() throws Exception {
@Test
public void testRenameRecoveryUnsupportedForFlatNamespace() throws Exception {
Assume.assumeTrue(!isNamespaceEnabled);
+ // In DFS endpoint, renamePath is O(1) API call and idempotency issue can happen.
+ // For blob endpoint, client orchestrates the rename operation.
+ Assume.assumeTrue(getFileSystem().getAbfsStore().getClient() instanceof AbfsDfsClient);
Review Comment:
Now we have getAbfsServiceType() method in trunk so can be use everywhere where we are checking for client, we can directly check for service type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2623303017
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 12 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 48s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 41s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/12/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 11 new + 13 unchanged - 3 fixed = 24 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 25s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 34s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 131m 54s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 37768caeef6a 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c3593befb893cf81147f8b2c93b060b6aa0c2529 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/12/testReport/ |
| Max. process+thread count | 568 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937114150
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1114,15 +1129,14 @@ public boolean rename(final Path source,
do {
try (AbfsPerfInfo perfInfo = startTracking("rename", "renamePath")) {
- boolean isNamespaceEnabled = getIsNamespaceEnabled(tracingContext);
final AbfsClientRenameResult abfsClientRenameResult =
getClient().renamePath(sourceRelativePath, destinationRelativePath,
- continuation, tracingContext, sourceEtag, false,
- isNamespaceEnabled);
+ continuation, tracingContext, sourceEtag, false);
AbfsRestOperation op = abfsClientRenameResult.getOp();
perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ continuation = op.getResult()
Review Comment:
Earlier the formating was not correct, so I thought to correct in rename flow. That why this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2631619790
------------------------------
:::: AGGREGATED TEST RESULT ::::
============================================================
HNS-OAuth
============================================================
[WARNING] Tests run: 161, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 758, Failures: 0, Errors: 0, Skipped: 144
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23
============================================================
HNS-SharedKey
============================================================
[WARNING] Tests run: 161, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 758, Failures: 0, Errors: 0, Skipped: 96
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 10
============================================================
NonHNS-SharedKey
============================================================
[WARNING] Tests run: 161, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 742, Failures: 0, Errors: 0, Skipped: 339
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 11
============================================================
AppendBlob-HNS-OAuth
============================================================
[WARNING] Tests run: 161, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 758, Failures: 0, Errors: 0, Skipped: 149
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 49
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23
Time taken: 99 mins 10 secs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916365565
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
Review Comment:
COLON constant can be used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2572979708
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 12m 0s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 54s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/4/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 21 new + 13 unchanged - 3 fixed = 34 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 16s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 20s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 125m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 90d55ba03290 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e8c17ac188fdf076bfb70b4324134f8b255e4f29 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/4/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936110314
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1013,10 +1035,10 @@ public OutputStream openFileForWrite(final Path path,
if (getClient().checkIsDir(op.getResult())) {
throw new AbfsRestOperationException(
- AzureServiceErrorCode.PATH_NOT_FOUND.getStatusCode(),
- AzureServiceErrorCode.PATH_NOT_FOUND.getErrorCode(),
- "openFileForWrite must be used with files and not directories",
- null);
+ AzureServiceErrorCode.PATH_NOT_FOUND.getStatusCode(),
Review Comment:
at multiple places, formatting changes have come as part of latest iteration, which can be reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936662829
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/SASTokenProvider.java:
##########
@@ -49,6 +49,9 @@ public interface SASTokenProvider {
String SET_PERMISSION_OPERATION = "set-permission";
String SET_PROPERTIES_OPERATION = "set-properties";
String WRITE_OPERATION = "write";
+ String COPY_BLOB_DESTINATION = "copy-blob-dst";
Review Comment:
I can still see them, please remove
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -499,21 +526,36 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws AzureBlobFileSystemException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
- final String destination,
- final String continuation,
- final TracingContext tracingContext,
- final String sourceEtag,
- final boolean isMetadataIncompleteState,
- final boolean isNamespaceEnabled) throws IOException {
- /**
- * TODO: [FnsOverBlob] To be implemented as part of rename-delete over blob endpoint work. <a href="https://issues.apache.org/jira/browse/HADOOP-19233">HADOOP-19233</a>.
- */
- throw new NotImplementedException("Rename operation on Blob endpoint yet to be implemented.");
+ final String destination,
+ final String continuation,
+ final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
+ throws IOException {
+ BlobRenameHandler blobRenameHandler = getBlobRenameHandler(source,
+ destination, sourceEtag, isAtomicRenameKey(source), tracingContext
+ );
+ incrementAbfsRenamePath();
+ return blobRenameHandler.execute();
Review Comment:
null check still there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933372250
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java:
##########
@@ -227,6 +236,13 @@ public void testDeleteIdempotency() throws Exception {
public void testDeleteIdempotencyTriggerHttp404() throws Exception {
final AzureBlobFileSystem fs = getFileSystem();
+ /*
+ * Delete call for a folder on DFS endpoint is one-server call and the
+ * orchestration of delete of paths inside the directory. For Blob
+ * endpoint, the orchestration would be done by the client. The idempotency
+ * issue would not happen for blob endpoint.
+ */
+ assertTrue(fs.getAbfsClient() instanceof AbfsDfsClient);
Review Comment:
Yes checked now, this change will come as part of ingress PR
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java:
##########
@@ -314,4 +347,278 @@ public void deleteBlobDirParallelThreadToDeleteOnDifferentTracingContext()
fs.delete(new Path("/testDir"), true);
fs.close();
}
+
+ private void assumeBlobClient() throws IOException {
Review Comment:
It is available in AbstractAbfsIntegrationTest class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1910278262
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/HttpHeaderConfigurations.java:
##########
@@ -104,5 +104,9 @@ public final class HttpHeaderConfigurations {
*/
public static final String X_MS_BLOB_CONTENT_MD5 = "x-ms-blob-content-md5";
+ public static final String X_MS_COPY_ID = "x-ms-copy-id";
Review Comment:
javadocs for new constants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2579470691
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 1s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/7/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 4 new + 13 unchanged - 3 fixed = 17 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 39s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 134m 43s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 79efdf0cd2bb 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6863cf8f2cec18c802c4648c8974bff7894ff1b4 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/7/testReport/ |
| Max. process+thread count | 535 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1912739077
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -499,21 +556,50 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws IOException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
- final String destination,
- final String continuation,
- final TracingContext tracingContext,
- final String sourceEtag,
- final boolean isMetadataIncompleteState,
- final boolean isNamespaceEnabled) throws IOException {
- /**
- * TODO: [FnsOverBlob] To be implemented as part of rename-delete over blob endpoint work. <a href="https://issues.apache.org/jira/browse/HADOOP-19233">HADOOP-19233</a>.
- */
- throw new NotImplementedException("Rename operation on Blob endpoint yet to be implemented.");
+ final String destination,
+ final String continuation,
+ final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
+ throws IOException {
+ BlobRenameHandler blobRenameHandler = getBlobRenameHandler(source,
+ destination, sourceEtag, isAtomicRenameKey(source), tracingContext
+ );
+ incrementAbfsRenamePath();
+ if (blobRenameHandler.execute()) {
+ final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
+ final URL url = createRequestUrl(destination, abfsUriQueryBuilder.toString());
+ final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
+ final AbfsRestOperation successOp = getAbfsRestOperation(
+ AbfsRestOperationType.RenamePath, HTTP_METHOD_PUT,
+ url, requestHeaders);
+ successOp.hardSetResult(HttpURLConnection.HTTP_OK);
+ return new AbfsClientRenameResult(successOp, true, false);
+ } else {
+ throw new AbfsRestOperationException(HTTP_INTERNAL_ERROR,
+ AzureServiceErrorCode.UNKNOWN.getErrorCode(),
+ "FNS-Blob Rename was not successfull",
Review Comment:
Should add path or file name as well where rename failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1912737237
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/AzureServiceErrorCode.java:
##########
@@ -45,6 +45,8 @@ public enum AzureServiceErrorCode {
INVALID_SOURCE_OR_DESTINATION_RESOURCE_TYPE("InvalidSourceOrDestinationResourceType", HttpURLConnection.HTTP_CONFLICT, null),
RENAME_DESTINATION_PARENT_PATH_NOT_FOUND("RenameDestinationParentPathNotFound", HttpURLConnection.HTTP_NOT_FOUND, null),
INVALID_RENAME_SOURCE_PATH("InvalidRenameSourcePath", HttpURLConnection.HTTP_CONFLICT, null),
+ DIRECTORY_NOT_EMPTY_DELETE("DirectoryNotEmpty", HttpURLConnection.HTTP_CONFLICT,
Review Comment:
naming could be changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916358749
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
Review Comment:
javadocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916525068
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
+//import java.util.Collections;
+//import java.util.List;
+import java.util.Random;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsDriverException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.contracts.services.BlobAppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static java.net.HttpURLConnection.HTTP_PRECON_FAILED;
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.BLOCK_ID_LENGTH;
+//import static org.apache.hadoop.fs.azurebfs.services.AzureIngressHandler.generateBlockListXml;
+
+/**
+ * For a directory enabled for atomic-rename, before rename starts, a file with
+ * -RenamePending.json suffix is created. In this file, the states required for the
+ * rename operation are given. This file is created by {@link #preRename()} method.
+ * This is important in case the JVM process crashes during rename, the atomicity
+ * will be maintained, when the job calls {@link AzureBlobFileSystem#listStatus(Path)}
+ * or {@link AzureBlobFileSystem#getFileStatus(Path)}. On these API calls to filesystem,
+ * it will be checked if there is any RenamePending JSON file. If yes, the crashed rename
+ * operation would be resumed as per the file.
+ */
+public class RenameAtomicity {
+
+ private final TracingContext tracingContext;
+
+ private Path src, dst;
+
+ private String srcEtag;
+
+ private final AbfsBlobClient abfsClient;
+
+ private final Path renameJsonPath;
+
+ public static final String SUFFIX = "-RenamePending.json";
+
+ private int preRenameRetryCount = 0;
+
+ private int renamePendingJsonLen;
+
+ private final AbfsLease sourcePathLease;
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private static final Random RANDOM = new Random();
+
+ /**
+ * Performs pre-rename operations. Creates a file with -RenamePending.json
+ * suffix in the source parent directory. This file contains the states
+ * required for the rename operation.
+ *
+ * @param src Source path
+ * @param dst Destination path
+ * @param renameJsonPath Path of the JSON file to be created
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ */
+ public RenameAtomicity(final Path src, final Path dst,
+ final Path renameJsonPath,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient) {
+ this.src = src;
+ this.dst = dst;
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.sourcePathLease = null;
+ }
+
+ /**
+ * Resumes the rename operation from the JSON file.
+ *
+ * @param renameJsonPath Path of the JSON file
+ * @param renamePendingJsonFileLen Length of the JSON file
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ * @param sourceLease Lease of the source directory
+ */
+ public RenameAtomicity(final Path renameJsonPath,
+ final int renamePendingJsonFileLen,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient,
+ final AbfsLease sourceLease) {
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.renamePendingJsonLen = renamePendingJsonFileLen;
+ this.sourcePathLease = sourceLease;
+ }
+
+ /**
+ * Redo the rename operation from the JSON file.
+ *
+ * @throws AzureBlobFileSystemException If the redo operation fails.
+ */
+ public void redo() throws AzureBlobFileSystemException {
+ byte[] buffer = readRenamePendingJson(renameJsonPath, renamePendingJsonLen);
+ String contents = new String(buffer, Charset.defaultCharset());
+ try {
+ final RenamePendingJsonFormat renamePendingJsonFormatObj;
+ try {
+ renamePendingJsonFormatObj = objectMapper.readValue(contents,
+ RenamePendingJsonFormat.class);
+ } catch (JsonProcessingException e) {
+ return;
+ }
+ if (renamePendingJsonFormatObj != null && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getOldFolderName())
+ && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getNewFolderName())
+ && StringUtils.isNotEmpty(renamePendingJsonFormatObj.getETag())) {
+ this.src = new Path(renamePendingJsonFormatObj.getOldFolderName());
+ this.dst = new Path(renamePendingJsonFormatObj.getNewFolderName());
+ this.srcEtag = renamePendingJsonFormatObj.getETag();
+
+ BlobRenameHandler blobRenameHandler = new BlobRenameHandler(
+ this.src.toUri().getPath(), dst.toUri().getPath(),
+ abfsClient, srcEtag, true, true,
+ sourcePathLease, tracingContext);
+
+ blobRenameHandler.execute();
+ }
+ } finally {
+ deleteRenamePendingJson();
+ }
+ }
+
+ /** Read the JSON file.
+ *
+ * @param path Path of the JSON file
+ * @param len Length of the JSON file
+ * @return Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the read operation fails.
+ */
+ @VisibleForTesting
+ byte[] readRenamePendingJson(Path path, int len)
+ throws AzureBlobFileSystemException {
+ byte[] bytes = new byte[len];
+ abfsClient.read(path.toUri().getPath(), 0, bytes, 0,
+ len, null, null, null,
+ tracingContext);
+ return bytes;
+ }
+
+ /** Generate a random block ID.
+ *
+ * @return Random block ID
+ */
+ public static String generateBlockId() {
+ // PutBlock on the path.
+ byte[] blockIdByteArray = new byte[BLOCK_ID_LENGTH];
+ RANDOM.nextBytes(blockIdByteArray);
+ return new String(Base64.encodeBase64(blockIdByteArray),
+ StandardCharsets.UTF_8);
+ }
+
+ /** Create the JSON file with the contents.
+ *
+ * @param path Path of the JSON file
+ * @param bytes Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the create operation fails.
+ */
+ @VisibleForTesting
+ void createRenamePendingJson(Path path, byte[] bytes)
+ throws AzureBlobFileSystemException {
+ // PutBlob on the path.
+ AbfsRestOperation putBlobOp = abfsClient.createPath(path.toUri().getPath(),
+ true,
+ true,
+ null,
+ false,
+ null,
+ null,
+ tracingContext);
+ String eTag = extractEtagHeader(putBlobOp.getResult());
+
+ String blockId = generateBlockId();
+ AppendRequestParameters appendRequestParameters
+ = new AppendRequestParameters(0, 0,
+ bytes.length, AppendRequestParameters.Mode.APPEND_MODE, false, null,
+ abfsClient.getAbfsConfiguration().isExpectHeaderEnabled(),
+ new BlobAppendRequestParameters(blockId, eTag));
+
+ abfsClient.append(path.toUri().getPath(), bytes,
+ appendRequestParameters, null, null, tracingContext);
+
+// List<String> blockIdList = new ArrayList<>(Collections.singleton(blockId));
+// String blockList = generateBlockListXml(blockIdList);
+ // PutBlockList on the path.
+ String blockList = "";
Review Comment:
if flush is called on empty string, how does it take the blockId into usage ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933339471
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -518,17 +479,25 @@ private void copyPath(final Path src, final Path dst, final String leaseId)
tracingContext, null, false);
final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ src.toUri().getPath();
- if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
- getDstSource(dstPathStatus)))) {
+ if (dstPathStatus != null && dstPathStatus.getResult() != null
+ && (srcCopyPath.equals(getDstSource(dstPathStatus)))) {
return;
}
}
throw ex;
}
final long pollWait = getAbfsClient().getAbfsConfiguration()
.getBlobCopyProgressPollWaitMillis();
+ final long maxWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressMaxWaitMillis();
+ long startTime = System.currentTimeMillis();
while (handleCopyInProgress(dst, tracingContext, copyId)
== BlobCopyProgress.PENDING) {
+ if (System.currentTimeMillis() - startTime > maxWait) {
+ throw new TimeoutException(
+ "Blob copy progress wait time exceeded for source: "
Review Comment:
use {} format for messages instead of concat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936767371
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -609,10 +694,12 @@ public AbfsRestOperation acquireLease(final String path, final int duration,
public AbfsRestOperation renewLease(final String path, final String leaseId,
TracingContext tracingContext) throws AzureBlobFileSystemException {
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
- requestHeaders.add(new AbfsHttpHeader(X_MS_LEASE_ACTION, RENEW_LEASE_ACTION));
+ requestHeaders.add(
+ new AbfsHttpHeader(X_MS_LEASE_ACTION, RENEW_LEASE_ACTION));
requestHeaders.add(new AbfsHttpHeader(X_MS_LEASE_ID, leaseId));
- final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
+ final AbfsUriQueryBuilder abfsUriQueryBuilder
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936767608
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1118,11 +1135,14 @@ public boolean rename(final Path source,
final AbfsClientRenameResult abfsClientRenameResult =
getClient().renamePath(sourceRelativePath, destinationRelativePath,
continuation, tracingContext, sourceEtag, false,
- isNamespaceEnabled);
+ isNamespaceEnabled);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936796576
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
+//import java.util.Collections;
+//import java.util.List;
+import java.util.Random;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsDriverException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.contracts.services.BlobAppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static java.net.HttpURLConnection.HTTP_PRECON_FAILED;
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.BLOCK_ID_LENGTH;
+//import static org.apache.hadoop.fs.azurebfs.services.AzureIngressHandler.generateBlockListXml;
+
+/**
+ * For a directory enabled for atomic-rename, before rename starts, a file with
+ * -RenamePending.json suffix is created. In this file, the states required for the
+ * rename operation are given. This file is created by {@link #preRename()} method.
+ * This is important in case the JVM process crashes during rename, the atomicity
+ * will be maintained, when the job calls {@link AzureBlobFileSystem#listStatus(Path)}
+ * or {@link AzureBlobFileSystem#getFileStatus(Path)}. On these API calls to filesystem,
+ * it will be checked if there is any RenamePending JSON file. If yes, the crashed rename
+ * operation would be resumed as per the file.
+ */
+public class RenameAtomicity {
+
+ private final TracingContext tracingContext;
+
+ private Path src, dst;
+
+ private String srcEtag;
+
+ private final AbfsBlobClient abfsClient;
+
+ private final Path renameJsonPath;
+
+ public static final String SUFFIX = "-RenamePending.json";
+
+ private int preRenameRetryCount = 0;
+
+ private int renamePendingJsonLen;
+
+ private final AbfsLease sourcePathLease;
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private static final Random RANDOM = new Random();
+
+ /**
+ * Performs pre-rename operations. Creates a file with -RenamePending.json
+ * suffix in the source parent directory. This file contains the states
+ * required for the rename operation.
+ *
+ * @param src Source path
+ * @param dst Destination path
+ * @param renameJsonPath Path of the JSON file to be created
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ */
+ public RenameAtomicity(final Path src, final Path dst,
+ final Path renameJsonPath,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient) {
+ this.src = src;
+ this.dst = dst;
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.sourcePathLease = null;
+ }
+
+ /**
+ * Resumes the rename operation from the JSON file.
+ *
+ * @param renameJsonPath Path of the JSON file
+ * @param renamePendingJsonFileLen Length of the JSON file
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ * @param sourceLease Lease of the source directory
+ */
+ public RenameAtomicity(final Path renameJsonPath,
+ final int renamePendingJsonFileLen,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient,
+ final AbfsLease sourceLease) {
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.renamePendingJsonLen = renamePendingJsonFileLen;
+ this.sourcePathLease = sourceLease;
+ }
+
+ /**
+ * Redo the rename operation from the JSON file.
+ *
+ * @throws AzureBlobFileSystemException If the redo operation fails.
+ */
+ public void redo() throws AzureBlobFileSystemException {
+ byte[] buffer = readRenamePendingJson(renameJsonPath, renamePendingJsonLen);
+ String contents = new String(buffer, Charset.defaultCharset());
+ try {
+ final RenamePendingJsonFormat renamePendingJsonFormatObj;
+ try {
+ renamePendingJsonFormatObj = objectMapper.readValue(contents,
+ RenamePendingJsonFormat.class);
+ } catch (JsonProcessingException e) {
+ return;
+ }
+ if (renamePendingJsonFormatObj != null && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getOldFolderName())
+ && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getNewFolderName())
+ && StringUtils.isNotEmpty(renamePendingJsonFormatObj.getETag())) {
+ this.src = new Path(renamePendingJsonFormatObj.getOldFolderName());
+ this.dst = new Path(renamePendingJsonFormatObj.getNewFolderName());
+ this.srcEtag = renamePendingJsonFormatObj.getETag();
+
+ BlobRenameHandler blobRenameHandler = new BlobRenameHandler(
+ this.src.toUri().getPath(), dst.toUri().getPath(),
+ abfsClient, srcEtag, true, true,
+ sourcePathLease, tracingContext);
+
+ blobRenameHandler.execute();
+ }
+ } finally {
+ deleteRenamePendingJson();
+ }
+ }
+
+ /** Read the JSON file.
+ *
+ * @param path Path of the JSON file
+ * @param len Length of the JSON file
+ * @return Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the read operation fails.
+ */
+ @VisibleForTesting
+ byte[] readRenamePendingJson(Path path, int len)
+ throws AzureBlobFileSystemException {
+ byte[] bytes = new byte[len];
+ abfsClient.read(path.toUri().getPath(), 0, bytes, 0,
+ len, null, null, null,
+ tracingContext);
+ return bytes;
+ }
+
+ /** Generate a random block ID.
+ *
+ * @return Random block ID
+ */
+ public static String generateBlockId() {
+ // PutBlock on the path.
+ byte[] blockIdByteArray = new byte[BLOCK_ID_LENGTH];
+ RANDOM.nextBytes(blockIdByteArray);
+ return new String(Base64.encodeBase64(blockIdByteArray),
+ StandardCharsets.UTF_8);
+ }
+
+ /** Create the JSON file with the contents.
+ *
+ * @param path Path of the JSON file
+ * @param bytes Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the create operation fails.
+ */
+ @VisibleForTesting
+ void createRenamePendingJson(Path path, byte[] bytes)
+ throws AzureBlobFileSystemException {
+ // PutBlob on the path.
+ AbfsRestOperation putBlobOp = abfsClient.createPath(path.toUri().getPath(),
+ true,
+ true,
+ null,
+ false,
+ null,
+ null,
+ tracingContext);
+ String eTag = extractEtagHeader(putBlobOp.getResult());
+
+ String blockId = generateBlockId();
+ AppendRequestParameters appendRequestParameters
+ = new AppendRequestParameters(0, 0,
+ bytes.length, AppendRequestParameters.Mode.APPEND_MODE, false, null,
+ abfsClient.getAbfsConfiguration().isExpectHeaderEnabled(),
+ new BlobAppendRequestParameters(blockId, eTag));
+
+ abfsClient.append(path.toUri().getPath(), bytes,
+ appendRequestParameters, null, null, tracingContext);
+
+// List<String> blockIdList = new ArrayList<>(Collections.singleton(blockId));
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2626908127
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 13 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 28s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/22/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 26s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/22/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 36s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 116m 16s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/22/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 7866961c0efa 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 689b653a78391ae44c38387212e0bf926209990c |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/22/testReport/ |
| Max. process+thread count | 701 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/22/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2627348647
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 14 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 44s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 37s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 25s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 40s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 130m 30s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/24/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 56040149e4a9 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e3f257a099bc644bd87cfa9e327645efb15fca8f |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/24/testReport/ |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/24/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2627698455
============================================================
HNS-OAuth-DFS
============================================================
[WARNING] Tests run: 160, Failures: 0, Errors: 0, Skipped: 3
[ERROR] Tests run: 758, Failures: 0, Errors: 0, Skipped: 142
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23
============================================================
HNS-SharedKey-DFS
============================================================
[WARNING] Tests run: 160, Failures: 0, Errors: 0, Skipped: 4
[ERROR] Tests run: 758, Failures: 0, Errors: 0, Skipped: 94
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 25
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 10
============================================================
NonHNS-SharedKey-DFS
============================================================
[WARNING] Tests run: 160, Failures: 0, Errors: 0, Skipped: 11
[ERROR] Tests run: 742, Failures: 0, Errors: 0, Skipped: 340
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 11
============================================================
AppendBlob-HNS-OAuth-DFS
============================================================
[WARNING] Tests run: 160, Failures: 0, Errors: 0, Skipped: 3
[ERROR] Tests run: 758, Failures: 0, Errors: 0, Skipped: 147
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 49
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 23
============================================================
NonHNS-OAuth-DFS
============================================================
[WARNING] Tests run: 160, Failures: 0, Errors: 0, Skipped: 11
[ERROR] Tests run: 742, Failures: 0, Errors: 0, Skipped: 344
[WARNING] Tests run: 171, Failures: 0, Errors: 0, Skipped: 27
[WARNING] Tests run: 262, Failures: 0, Errors: 0, Skipped: 24
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1910279797
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -359,6 +375,34 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
return op;
}
+ private void fixAtomicEntriesInListResults(final AbfsRestOperation op,
Review Comment:
add javadocs for all methods wherever missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1912744844
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1318,6 +1421,128 @@ public static String getDirectoryQueryParameter(final String path) {
return directory;
}
+ public boolean isAtomicRenameKey(String key) {
+ return isKeyForDirectorySet(key, azureAtomicRenameDirSet);
+ }
+
+ /**
+ * Action to be taken when atomic-key is present on a getPathStatus path.
+ *
+ * @param path path of the pendingJson for the atomic path.
+ * @param pathLease lease on the path.
+ * @param tracingContext tracing context.
+ *
+ * @throws AzureBlobFileSystemException server error or the path is renamePending json file and action is taken.
+ */
+ public void takeGetPathStatusAtomicRenameKeyAction(final Path path,
+ final AbfsLease pathLease,
+ final TracingContext tracingContext)
+ throws AzureBlobFileSystemException {
+ if (path == null || path.isRoot() || !isAtomicRenameKey(path.toUri().getPath())) {
+ return;
+ }
+ AbfsRestOperation pendingJsonFileStatus;
+ Path pendingJsonPath = new Path(path.getParent(),
+ path.toUri().getPath() + RenameAtomicity.SUFFIX);
+ try {
+ pendingJsonFileStatus = getPathStatus(
+ pendingJsonPath.toUri().getPath(), tracingContext, null, false);
+ if (checkIsDir(pendingJsonFileStatus.getResult())) {
+ return;
+ }
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_NOT_FOUND) {
+ return;
+ }
+ throw ex;
+ }
+
+ boolean renameSrcHasChanged;
+ try {
+ RenameAtomicity renameAtomicity = getRedoRenameAtomicity(
+ pendingJsonPath, Integer.parseInt(pendingJsonFileStatus.getResult()
+ .getResponseHeader(HttpHeaderConfigurations.CONTENT_LENGTH)),
+ tracingContext, pathLease);
+ renameAtomicity.redo();
+ renameSrcHasChanged = false;
+ } catch (AbfsRestOperationException ex) {
+ /*
+ * At this point, the source marked by the renamePending json file, might have
+ * already got renamed by some parallel thread, or at this point, the path
+ * would have got modified which would result in eTag change, which would lead
+ * to a HTTP_CONFLICT. In this case, no more operation needs to be taken, and
+ * the calling getPathStatus can return this source path as result.
+ */
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_NOT_FOUND
+ || ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ renameSrcHasChanged = true;
+ } else {
+ throw ex;
+ }
+ }
+ if (!renameSrcHasChanged) {
+ throw new AbfsRestOperationException(
+ AzureServiceErrorCode.PATH_NOT_FOUND.getStatusCode(),
+ AzureServiceErrorCode.PATH_NOT_FOUND.getErrorCode(),
+ ATOMIC_DIR_RENAME_RECOVERY_ON_GET_PATH_EXCEPTION,
+ null);
+ }
+ }
+
+ /**
+ * Action to be taken when atomic-key is present on a listPath path.
+ *
+ * @param path path of the pendingJson for the atomic path.
+ * @param renamePendingJsonLen length of the pendingJson file.
+ * @param tracingContext tracing context.
+ *
+ * @return true if action is taken.
+ * @throws AzureBlobFileSystemException server error
+ */
+ private boolean takeListPathAtomicRenameKeyAction(final Path path,
+ final int renamePendingJsonLen,
+ final TracingContext tracingContext)
+ throws AzureBlobFileSystemException {
+ if (path == null || path.isRoot() || !isAtomicRenameKey(
+ path.toUri().getPath()) || !path.toUri()
+ .getPath()
+ .endsWith(RenameAtomicity.SUFFIX)) {
+ return false;
+ }
+ try {
+ RenameAtomicity renameAtomicity
+ = getRedoRenameAtomicity(path, renamePendingJsonLen,
+ tracingContext, null);
+ renameAtomicity.redo();
+ } catch (AbfsRestOperationException ex) {
+ /*
+ * At this point, the source marked by the renamePending json file, might have
+ * already got renamed by some parallel thread, or at this point, the path
+ * would have got modified which would result in eTag change, which would lead
+ * to a HTTP_CONFLICT. In this case, no more operation needs to be taken, but
+ * since this is a renamePendingJson file and would be deleted by the redo operation,
+ * the calling listPath should not return this json path as result.
+ */
+ if (ex.getStatusCode() != HttpURLConnection.HTTP_NOT_FOUND
+ && ex.getStatusCode() != HttpURLConnection.HTTP_CONFLICT) {
+ throw ex;
+ }
+ }
+ return true;
Review Comment:
why are we directly returning true here and not checking for renameSrchasChanged as in the previous method ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936102119
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemCreate.java:
##########
@@ -171,50 +169,130 @@ public void testCreateNonRecursive2() throws Exception {
assertIsFile(fs, testFile);
}
+ private void assumeNonHnsAccountBlobEndpoint(final AzureBlobFileSystem fs) {
+ Assume.assumeTrue(fs.getAbfsStore().getClient() instanceof AbfsBlobClient);
+ }
+
+ /**
+ * Test createNonRecursive when parent exist.
+ *
+ * @throws Exception in case of failure
+ */
+ @Test
+ public void testCreateNonRecursiveWhenParentExist() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path createDirectoryPath = new Path("hbase/A");
+ fs.mkdirs(createDirectoryPath);
+ fs.createNonRecursive(new Path(createDirectoryPath, "B"), FsPermission
+ .getDefault(), false, 1024, (short) 1, 1024, null);
+ Assertions.assertThat(fs.exists(new Path(createDirectoryPath, "B"))).isTrue();
Review Comment:
Add assertion message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2625026516
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 33s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 29s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 14 unchanged - 3 fixed = 14 total (was 17) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 25s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 129m 54s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/18/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 60626bfb5cb8 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 52069e183c04c3ccd6f365fbd6e1981cb985199f |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/18/testReport/ |
| Max. process+thread count | 610 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/18/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933353546
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -518,17 +479,25 @@ private void copyPath(final Path src, final Path dst, final String leaseId)
tracingContext, null, false);
final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ src.toUri().getPath();
- if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
- getDstSource(dstPathStatus)))) {
+ if (dstPathStatus != null && dstPathStatus.getResult() != null
+ && (srcCopyPath.equals(getDstSource(dstPathStatus)))) {
return;
}
}
throw ex;
}
final long pollWait = getAbfsClient().getAbfsConfiguration()
.getBlobCopyProgressPollWaitMillis();
+ final long maxWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressMaxWaitMillis();
+ long startTime = System.currentTimeMillis();
while (handleCopyInProgress(dst, tracingContext, copyId)
== BlobCopyProgress.PENDING) {
+ if (System.currentTimeMillis() - startTime > maxWait) {
+ throw new TimeoutException(
+ "Blob copy progress wait time exceeded for source: "
Review Comment:
Replaced it with String.format()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933348835
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -415,6 +415,16 @@ private void fixAtomicEntriesInListResults(final AbfsRestOperation op,
listResultSchema.withPaths(filteredEntries);
}
+ /**{@inheritDoc}*/
+ @Override
+ public void createNonRecursivePreCheck(Path parentPath, TracingContext tracingContext)
+ throws IOException {
Review Comment:
ParentPath as null is only possible when f is root directory. So, check for this is already covered in the caller method. But for safe side I have added null check on parentPath in this method as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931661909
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -192,10 +239,1304 @@ public void testRenameWithNoDestinationParentDir() throws Exception {
// Verify that metadata was in an incomplete state after the rename
// failure, and we retired the rename once more.
IOStatistics ioStatistics = fs.getIOStatistics();
+ AbfsClient client = fs.getAbfsStore().getClient();
IOStatisticAssertions.assertThatStatisticCounter(ioStatistics,
RENAME_PATH_ATTEMPTS.getStatName())
- .describedAs("There should be 2 rename attempts if metadata "
- + "incomplete state failure is hit")
- .isEqualTo(2);
+ .describedAs("For Dfs endpoint: There should be 2 rename "
+ + "attempts if metadata incomplete state failure is hit."
+ + "For Blob endpoint: There would be only one rename attempt which "
+ + "would have a failed precheck.")
+ .isEqualTo(client instanceof AbfsDfsClient ? 2 : 1);
+ }
+
+ @Test
+ public void testRenameToRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ fs.mkdirs(new Path("/src1/src2"));
+ assertTrue(fs.rename(new Path("/src1/src2"), new Path("/")));
+ assertTrue(fs.exists(new Path("/src2")));
+ }
+
+ @Test
+ public void testRenameNotFoundBlobToEmptyRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assertFalse(fs.rename(new Path("/file"), new Path("/")));
+ }
+
+ private void assumeNonHnsAccountBlobEndpoint(final AzureBlobFileSystem fs) {
+ assertTrue(fs.getAbfsStore().getClient() instanceof AbfsBlobClient);
+ }
+
+ @Test(expected = IOException.class)
+ public void testRenameBlobToDstWithColonInPath() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.create(new Path("/src"));
+ fs.rename(new Path("/src"), new Path("/dst:file"));
+ }
+
+ @Test
+ public void testRenameBlobInSameDirectoryWithNoMarker() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) fs.getAbfsStore().getClient();
+ fs.create(new Path("/srcDir/dir/file"));
+ client.deleteBlobPath(new Path("/srcDir/dir"), null,
+ getTestTracingContext(fs, true));
+ assertTrue(fs.rename(new Path("/srcDir/dir"), new Path("/srcDir")));
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWhereDirectoryAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.mkdirs(new Path("testDir2/test4/test3"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3")));
+ if (getIsNamespaceEnabled(fs)
+ || fs.getAbfsClient() instanceof AbfsBlobClient) {
+ assertFalse(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ } else {
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ }
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWherePartAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.create(new Path("testDir2/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("testDir2/test4/"));
+ fs.create(new Path("testDir2/test4/file1"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ Assert.assertTrue(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3")));
+ assertFalse(fs.exists(new Path("testDir2/test4/file")));
+ assertTrue(fs.exists(new Path("testDir2/test4/file1")));
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file1")));
+ assertTrue(fs.exists(new Path("testDir2/test4/file1")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ }
+
+ /**
+ * Test that after completing rename for a directory which is enabled for
+ * AtomicRename, the RenamePending JSON file is deleted.
+ */
+ @Test
+ public void testRenamePendingJsonIsRemovedPostSuccessfulRename()
+ throws Exception {
+ final AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) addSpyHooksOnClient(fs);
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.setWorkingDirectory(new Path("/"));
+ fs.mkdirs(new Path("hbase/test1/test2/test3"));
+ fs.create(new Path("hbase/test1/test2/test3/file"));
+ fs.create(new Path("hbase/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("hbase/test4/"));
+ fs.create(new Path("hbase/test4/file1"));
+ final Integer[] correctDeletePathCount = new Integer[1];
+ correctDeletePathCount[0] = 0;
+ Mockito.doAnswer(answer -> {
+ final String correctDeletePath = "/hbase/test1/test2/test3" + SUFFIX;
+ if (correctDeletePath.equals(
+ ((Path) answer.getArgument(0)).toUri().getPath())) {
+ correctDeletePathCount[0] = 1;
+ }
+ return null;
+ })
+ .when(client)
+ .deleteBlobPath(Mockito.any(Path.class), Mockito.nullable(String.class),
+ Mockito.any(TracingContext.class));
+ assertTrue(fs.rename(new Path("hbase/test1/test2/test3"),
+ new Path("hbase/test4")));
+ assertEquals("RenamePendingJson should be deleted",
+ 1,
+ (int) correctDeletePathCount[0]);
+ }
+
+ private AbfsClient addSpyHooksOnClient(final AzureBlobFileSystem fs) {
+ AzureBlobFileSystemStore store = Mockito.spy(fs.getAbfsStore());
+ Mockito.doReturn(store).when(fs).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+ return client;
+ }
+
+ /**
+ * Test for a directory in /hbase directory. To simulate the crash of process,
+ * test will throw an exception with 403 on a copy of one of the blob.<br>
+ * ListStatus API will be called on the directory. Expectation is that the ListStatus
+ * API of {@link AzureBlobFileSystem} should recover the paused rename.
+ */
+ @Test
+ public void testHBaseHandlingForFailedRenameWithListRecovery()
+ throws Exception {
+ AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) addSpyHooksOnClient(fs);
+ String srcPath = "hbase/test1/test2";
+ final String failedCopyPath = srcPath + "/test3/file1";
+ fs.setWorkingDirectory(new Path("/"));
+ fs.mkdirs(new Path(srcPath));
+ fs.mkdirs(new Path(srcPath, "test3"));
+ fs.create(new Path(srcPath + "/test3/file"));
+ fs.create(new Path(failedCopyPath));
+ fs.mkdirs(new Path("hbase/test4/"));
+ fs.create(new Path("hbase/test4/file1"));
+ crashRenameAndRecover(fs, client, srcPath, (abfsFs) -> {
+ abfsFs.listStatus(new Path(srcPath).getParent());
+ return null;
+ });
+ }
+
+ /**
+ * Test for a directory in /hbase directory. To simulate the crash of process,
+ * test will throw an exception with 403 on a copy of one of the blob. The
+ * source directory is a nested directory.<br>
+ * GetFileStatus API will be called on the directory. Expectation is that the
+ * GetFileStatus API of {@link AzureBlobFileSystem} should recover the paused
+ * rename.
+ */
+ @Test
+ public void testHBaseHandlingForFailedRenameWithGetFileStatusRecovery()
+ throws Exception {
+ AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) addSpyHooksOnClient(fs);
+ String srcPath = "hbase/test1/test2";
+ final String failedCopyPath = srcPath + "/test3/file1";
+ fs.setWorkingDirectory(new Path("/"));
+ fs.mkdirs(new Path(srcPath));
+ fs.mkdirs(new Path(srcPath, "test3"));
+ fs.create(new Path(srcPath + "/test3/file"));
+ fs.create(new Path(failedCopyPath));
+ fs.mkdirs(new Path("hbase/test4/"));
+ fs.create(new Path("hbase/test4/file1"));
+ crashRenameAndRecover(fs, client, srcPath, (abfsFs) -> {
+ abfsFs.exists(new Path(srcPath));
+ return null;
+ });
+ }
+
+ private void crashRenameAndRecover(final AzureBlobFileSystem fs,
+ AbfsBlobClient client,
+ final String srcPath,
+ final FunctionRaisingIOE<AzureBlobFileSystem, Void> recoveryCallable)
+ throws Exception {
+ crashRename(fs, client, srcPath);
+ AzureBlobFileSystem fs2 = Mockito.spy(getFileSystem());
+ fs2.setWorkingDirectory(new Path(ROOT_PATH));
+ client = (AbfsBlobClient) addSpyHooksOnClient(fs2);
+ int[] renameJsonDeleteCounter = new int[1];
+ Mockito.doAnswer(answer -> {
+ if ((ROOT_PATH + srcPath + SUFFIX)
+ .equalsIgnoreCase(((Path) answer.getArgument(0)).toUri().getPath())) {
+ renameJsonDeleteCounter[0] = 1;
+ }
+ return answer.callRealMethod();
+ })
+ .when(client)
+ .deleteBlobPath(Mockito.any(Path.class), Mockito.nullable(String.class),
+ Mockito.any(TracingContext.class));
+ recoveryCallable.apply(fs2);
+ Assertions.assertThat(renameJsonDeleteCounter[0])
+ .describedAs("RenamePendingJson should be deleted")
+ .isEqualTo(1);
+ //List would complete the rename orchestration.
+ assertFalse(fs2.exists(new Path("hbase/test1/test2")));
+ assertFalse(fs2.exists(new Path("hbase/test1/test2/test3")));
+ assertTrue(fs2.exists(new Path("hbase/test4/test2/test3")));
+ assertFalse(fs2.exists(new Path("hbase/test1/test2/test3/file")));
+ assertTrue(fs2.exists(new Path("hbase/test4/test2/test3/file")));
+ assertFalse(fs2.exists(new Path("hbase/test1/test2/test3/file1")));
+ assertTrue(fs2.exists(new Path("hbase/test4/test2/test3/file1")));
+ }
+
+ private void crashRename(final AzureBlobFileSystem fs,
+ final AbfsBlobClient client,
+ final String srcPath) throws Exception {
+ BlobRenameHandler[] blobRenameHandlers = new BlobRenameHandler[1];
+ AbfsClientTestUtil.mockGetRenameBlobHandler(client,
+ blobRenameHandler -> {
+ blobRenameHandlers[0] = blobRenameHandler;
+ return null;
+ });
+ //Fail rename orchestration on path hbase/test1/test2/test3/file1
+ Mockito.doThrow(new AbfsRestOperationException(HTTP_FORBIDDEN, "", "",
+ new Exception()))
+ .when(client)
+ .copyBlob(Mockito.any(Path.class), Mockito.any(Path.class),
+ Mockito.nullable(String.class),
+ Mockito.any(TracingContext.class));
+ LambdaTestUtils.intercept(AccessDeniedException.class, () -> {
+ fs.rename(new Path(srcPath),
+ new Path("hbase/test4"));
+ });
+ //Release all the leases taken by atomic rename orchestration
+ List<AbfsLease> leases = new ArrayList<>(blobRenameHandlers[0].getLeases());
+ for (AbfsLease lease : leases) {
+ lease.free();
+ }
+ }
+
+ /**
+ * Simulates a scenario where HMaster in Hbase starts up and executes listStatus
+ * API on the directory that has to be renamed by some other executor-machine.
+ * The scenario is that RenamePending JSON is created but before it could be
+ * appended, it has been opened by the HMaster. The HMaster will delete it. The
+ * machine doing rename would have to recreate the JSON file.
+ * ref: <a href="https://issues.apache.org/jira/browse/HADOOP-12678">issue</a>
+ */
+ @Test
+ public void testHbaseListStatusBeforeRenamePendingFileAppendedWithIngressOnBlob()
+ throws Exception {
+ final AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ testRenamePreRenameFailureResolution(fs);
+ testAtomicityRedoInvalidFile(fs);
+ }
+
+ private void testRenamePreRenameFailureResolution(final AzureBlobFileSystem fs)
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931751723
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
Review Comment:
Renamed it to getDstPathForBlob
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936707126
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -689,21 +789,53 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws IOException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
final String destination,
final String continuation,
final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
Review Comment:
It will need changes even in AbfsDfsClient and AbfsClient. This is not added by me recently. It was there from the begining.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1556,6 +1721,128 @@ public static String getDirectoryQueryParameter(final String path) {
return directory;
}
+ public boolean isAtomicRenameKey(String key) {
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936798963
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -499,21 +526,36 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws AzureBlobFileSystemException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
- final String destination,
- final String continuation,
- final TracingContext tracingContext,
- final String sourceEtag,
- final boolean isMetadataIncompleteState,
- final boolean isNamespaceEnabled) throws IOException {
- /**
- * TODO: [FnsOverBlob] To be implemented as part of rename-delete over blob endpoint work. <a href="https://issues.apache.org/jira/browse/HADOOP-19233">HADOOP-19233</a>.
- */
- throw new NotImplementedException("Rename operation on Blob endpoint yet to be implemented.");
+ final String destination,
+ final String continuation,
+ final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
+ throws IOException {
+ BlobRenameHandler blobRenameHandler = getBlobRenameHandler(source,
+ destination, sourceEtag, isAtomicRenameKey(source), tracingContext
+ );
+ incrementAbfsRenamePath();
+ return blobRenameHandler.execute();
Review Comment:
Removed null check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2626695060
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 13 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 29s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/19/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 26s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/19/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 131m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/19/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 2175575e05ac 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / da502bf08b5c6d3fbf5a718fcedbdaf5bd4d00ec |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/19/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/19/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936989900
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java:
##########
@@ -409,6 +412,24 @@ public AbfsRestOperation createPath(final String path,
return op;
}
+ public void createNonRecursivePreCheck(Path parentPath,
Review Comment:
inheritdoc annotation is missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2626834071
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 13 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 25s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| -1 :x: | javadoc | 0m 29s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/20/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 24s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/20/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 29s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 130m 52s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/20/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 9886b1dd0fdb 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ba9b0255a062f3aaba2720c24bad523a34611288 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/20/testReport/ |
| Max. process+thread count | 537 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/20/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936997768
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/UriUtils.java:
##########
@@ -230,6 +240,28 @@ private static String replacedUrl(String baseUrl, String oldString, String newSt
return sb.toString();
}
+ public static boolean isKeyForDirectorySet(String key, Set<String> dirSet) {
Review Comment:
javadocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937112476
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##########
@@ -21,25 +21,24 @@
import java.io.IOException;
import java.lang.reflect.Field;
-import org.apache.hadoop.classification.VisibleForTesting;
-import org.apache.hadoop.fs.azurebfs.constants.AbfsServiceType;
-import org.apache.hadoop.fs.azurebfs.services.FixedSASTokenProvider;
-import org.apache.hadoop.fs.azurebfs.constants.HttpOperationType;
-import org.apache.hadoop.fs.azurebfs.utils.MetricFormat;
-import org.apache.hadoop.util.Preconditions;
+import org.slf4j.Logger;
Review Comment:
Thank you for fixing import ordering in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916390231
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
+ getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
+ == BlobCopyProgress.PENDING) {
+ try {
+ Thread.sleep(pollWait);
+ } catch (InterruptedException ignored) {
+
+ }
+ }
+ }
+
+ /** Gets the source path of the copy operation.
+ *
+ * @param dstPathStatus server response for the GetBlobProperties API on the
+ * destination path.
+ *
+ * @return source path of the copy operation
+ */
+ private String getDstSource(final AbfsRestOperation dstPathStatus) {
+ try {
+ String responseHeader = dstPathStatus.getResult()
+ .getResponseHeader(X_MS_COPY_SOURCE);
+ if (responseHeader == null) {
+ return null;
+ }
+ return new URL(responseHeader).toURI().getPath();
+ } catch (URISyntaxException | MalformedURLException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Verifies if the blob copy is success or a failure or still in progress.
+ *
+ * @param dstPath path of the destination for the copying
+ * @param tracingContext object of tracingContext used for the tracing of the
+ * server calls.
+ * @param copyId id returned by server on the copy server-call. This id gets
+ * attached to blob and is returned by GetBlobProperties API on the destination.
+ *
+ * @return BlobCopyProgress indicating the status of the copy operation
+ *
+ * @throws AzureBlobFileSystemException exception returned in making server call
+ * for GetBlobProperties on the path. It can be thrown if the copyStatus is failure
+ * or is aborted.
+ */
+ @VisibleForTesting
+ public BlobCopyProgress handleCopyInProgress(final Path dstPath,
+ final TracingContext tracingContext,
+ final String copyId) throws AzureBlobFileSystemException {
+ AbfsRestOperation op = getAbfsClient().getPathStatus(dstPath.toUri().getPath(),
+ tracingContext, null, false);
+
+ if (op.getResult() != null && copyId.equals(
+ op.getResult().getResponseHeader(X_MS_COPY_ID))) {
+ final String copyStatus = op.getResult()
Review Comment:
check for response header is not null is needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933621962
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsPaginatedDelete.java:
##########
@@ -142,6 +142,7 @@ private void setTestUserConf(Configuration conf, String key, String value) {
*/
@Test
public void testRecursiveDeleteWithPagination() throws Exception {
+ assertTrue(getFileSystem().getAbfsStore().getClient() instanceof AbfsDfsClient);
Review Comment:
Yes, it is assume. I have fixed it in next iteration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917796440
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/extensions/SASTokenProvider.java:
##########
@@ -49,6 +49,9 @@ public interface SASTokenProvider {
String SET_PERMISSION_OPERATION = "set-permission";
String SET_PROPERTIES_OPERATION = "set-properties";
String WRITE_OPERATION = "write";
+ String COPY_BLOB_DESTINATION = "copy-blob-dst";
Review Comment:
It was not in use anywhere, so removed it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917835360
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -909,23 +959,32 @@ public AbfsRestOperation read(final String path,
}
/**
- * Orchestration for delete operation to be implemented.
+ * Get Rest Operation for API <a href = https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete></a>.
+ * Delete the file or directory at specified path.
* @param path to be deleted.
* @param recursive if the path is a directory, delete recursively.
* @param continuation to specify continuation token.
- * @param tracingContext for tracing the server calls.
- * @param isNamespaceEnabled specify if the namespace is enabled.
+ * @param tracingContext TracingContext instance to track identifiers
+ * @param isNamespaceEnabled whether namespace enabled account or not
* @return executed rest operation containing response from server.
* @throws AzureBlobFileSystemException if rest operation fails.
*/
@Override
public AbfsRestOperation deletePath(final String path,
- final boolean recursive,
- final String continuation,
- TracingContext tracingContext,
- final boolean isNamespaceEnabled) throws AzureBlobFileSystemException {
- // TODO: [FnsOverBlob][HADOOP-19233] To be implemented as part of rename-delete over blob endpoint work.
- throw new NotImplementedException("Delete operation on Blob endpoint will be implemented in future.");
+ final boolean recursive,
+ final String continuation,
+ final TracingContext tracingContext,
+ final boolean isNamespaceEnabled) throws AzureBlobFileSystemException {
+ getBlobDeleteHandler(path, recursive, tracingContext).execute();
+ return null;
Review Comment:
Done the changes to return dummy response same as rename Path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1919824616
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -192,10 +239,1304 @@ public void testRenameWithNoDestinationParentDir() throws Exception {
// Verify that metadata was in an incomplete state after the rename
// failure, and we retired the rename once more.
IOStatistics ioStatistics = fs.getIOStatistics();
+ AbfsClient client = fs.getAbfsStore().getClient();
IOStatisticAssertions.assertThatStatisticCounter(ioStatistics,
RENAME_PATH_ATTEMPTS.getStatName())
- .describedAs("There should be 2 rename attempts if metadata "
- + "incomplete state failure is hit")
- .isEqualTo(2);
+ .describedAs("For Dfs endpoint: There should be 2 rename "
+ + "attempts if metadata incomplete state failure is hit."
+ + "For Blob endpoint: There would be only one rename attempt which "
+ + "would have a failed precheck.")
+ .isEqualTo(client instanceof AbfsDfsClient ? 2 : 1);
+ }
+
+ @Test
+ public void testRenameToRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ fs.mkdirs(new Path("/src1/src2"));
+ assertTrue(fs.rename(new Path("/src1/src2"), new Path("/")));
+ assertTrue(fs.exists(new Path("/src2")));
+ }
+
+ @Test
+ public void testRenameNotFoundBlobToEmptyRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assertFalse(fs.rename(new Path("/file"), new Path("/")));
+ }
+
+ private void assumeNonHnsAccountBlobEndpoint(final AzureBlobFileSystem fs) {
+ assertTrue(fs.getAbfsStore().getClient() instanceof AbfsBlobClient);
+ }
+
+ @Test(expected = IOException.class)
+ public void testRenameBlobToDstWithColonInPath() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.create(new Path("/src"));
+ fs.rename(new Path("/src"), new Path("/dst:file"));
+ }
+
+ @Test
+ public void testRenameBlobInSameDirectoryWithNoMarker() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) fs.getAbfsStore().getClient();
+ fs.create(new Path("/srcDir/dir/file"));
+ client.deleteBlobPath(new Path("/srcDir/dir"), null,
+ getTestTracingContext(fs, true));
+ assertTrue(fs.rename(new Path("/srcDir/dir"), new Path("/srcDir")));
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWhereDirectoryAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.mkdirs(new Path("testDir2/test4/test3"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3")));
+ if (getIsNamespaceEnabled(fs)
+ || fs.getAbfsClient() instanceof AbfsBlobClient) {
+ assertFalse(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ } else {
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ }
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWherePartAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.create(new Path("testDir2/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("testDir2/test4/"));
+ fs.create(new Path("testDir2/test4/file1"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ Assert.assertTrue(fs.rename(new Path("testDir2/test1/test2/test3"),
Review Comment:
use assertions.assertThat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933319435
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -415,6 +415,16 @@ private void fixAtomicEntriesInListResults(final AbfsRestOperation op,
listResultSchema.withPaths(filteredEntries);
}
+ /**{@inheritDoc}*/
+ @Override
+ public void createNonRecursivePreCheck(Path parentPath, TracingContext tracingContext)
+ throws IOException {
Review Comment:
parentPath can be null so we should have a null check for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933394315
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -38,6 +38,8 @@
import java.util.Map;
import java.util.TreeMap;
import java.util.UUID;
+import java.util.Arrays;
Review Comment:
Taken!
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobDeleteHandler.java:
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_CONFLICT;
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.NON_EMPTY_DIRECTORY_DELETE;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for delete over Blob endpoint. Blob endpoint for flat-namespace
+ * account does not support directory delete. This class is responsible for
+ * deleting the blobs and creating the parent directory marker file if needed.
+ */
+public class BlobDeleteHandler extends ListActionTaker {
+
+ private final Path path;
+
+ private final boolean recursive;
+
+ private boolean nonRecursiveDeleteDirectoryFailed = false;
+
+ private final TracingContext tracingContext;
+
+ private final AtomicInteger deleteCount = new AtomicInteger(0);
+
+ /** Constructor
+ *
+ * @param path path to delete.
+ * @param recursive if true, delete the path recursively.
+ * @param abfsBlobClient client to use for blob operations.
+ * @param tracingContext tracing context.
+ */
+ public BlobDeleteHandler(final Path path,
+ final boolean recursive,
+ final AbfsBlobClient abfsBlobClient,
+ final TracingContext tracingContext) {
+ super(path, abfsBlobClient, tracingContext);
+ this.path = path;
+ this.recursive = recursive;
+ this.tracingContext = tracingContext;
+ }
+
+ /**{@inheritDoc}
+ *
+ * @return the maximum number of parallelism for delete operation.
+ */
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobDeleteDirConsumptionParallelism();
+ }
+
+ /** Delete the path.
+ *
+ * @param path path to delete.
+ * @return true if the path is deleted.
+ * @throws AzureBlobFileSystemException server error.
+ */
+ private boolean deleteInternal(final Path path)
+ throws AzureBlobFileSystemException {
+ getAbfsClient().deleteBlobPath(path, null, tracingContext);
Review Comment:
We are maintaining this approach for consistency with other methods. We can revisit and decide later.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -499,21 +578,50 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws IOException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
- final String destination,
- final String continuation,
- final TracingContext tracingContext,
- final String sourceEtag,
- final boolean isMetadataIncompleteState,
- final boolean isNamespaceEnabled) throws IOException {
- /**
- * TODO: [FnsOverBlob] To be implemented as part of rename-delete over blob endpoint work. <a href="https://issues.apache.org/jira/browse/HADOOP-19233">HADOOP-19233</a>.
- */
- throw new NotImplementedException("Rename operation on Blob endpoint yet to be implemented.");
+ final String destination,
Review Comment:
Done the formatting as per the codestyle.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/AbfsHttpConstants.java:
##########
@@ -290,5 +290,13 @@ public static ApiVersion getCurrentVersion() {
public static final String JDK_FALLBACK = "JDK_fallback";
public static final String KEEP_ALIVE_CACHE_CLOSED = "KeepAliveCache is closed";
+ public static final String COPY_STATUS_SUCCESS = "success";
+ public static final String COPY_STATUS_PENDING = "pending";
+ public static final String COPY_STATUS_ABORTED = "aborted";
+ public static final String COPY_STATUS_FAILED = "failed";
+
+ public static final String ATOMIC_DIR_RENAME_RECOVERY_ON_GET_PATH_EXCEPTION =
Review Comment:
Taken!
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1124,8 +1144,11 @@ public void delete(final Path path, final boolean recursive,
try (AbfsPerfInfo perfInfo = startTracking("delete", "deletePath")) {
AbfsRestOperation op = getClient().deletePath(relativePath, recursive,
continuation, tracingContext, getIsNamespaceEnabled(tracingContext));
- perfInfo.registerResult(op.getResult());
- continuation = op.getResult().getResponseHeader(HttpHeaderConfigurations.X_MS_CONTINUATION);
+ if (op != null) {
Review Comment:
Same check as present for rename. This was discussed offline.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -909,23 +1025,46 @@ public AbfsRestOperation read(final String path,
}
/**
- * Orchestration for delete operation to be implemented.
+ * Get Rest Operation for API <a href = https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete></a>.
Review Comment:
Updated the comment.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java:
##########
@@ -45,10 +46,10 @@
* AbfsLease manages an Azure blob lease. It acquires an infinite lease on instantiation and
* releases the lease when free() is called. Use it to prevent writes to the blob by other
* processes that don't have the lease.
- *
+ * <p>
Review Comment:
It is not needed.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -68,9 +70,14 @@
import org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider;
import org.apache.hadoop.fs.azurebfs.security.ContextEncryptionAdapter;
import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.apache.hadoop.classification.VisibleForTesting;
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933349029
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -174,7 +151,7 @@ public boolean execute() throws AzureBlobFileSystemException {
if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
/*
* if it is not a resume of a previous failed atomic rename operation,
- * perform the pre-rename operation.
+ * Create the rename JSON.
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933352065
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -207,7 +207,7 @@ public void testPosixRenameDirectory() throws Exception {
Path testDir2 = path("testDir2");
fs.mkdirs(new Path(testDir2 + "/test1/test2/test3"));
fs.mkdirs(new Path(testDir2 + "/test4"));
- Assert.assertTrue(fs.rename(new Path(testDir2 + "/test1/test2/test3"), new Path(testDir2 + "/test4")));
+ assertTrue(fs.rename(new Path(testDir2 + "/test1/test2/test3"), new Path(testDir2 + "/test4")));
Review Comment:
use Assertions.assertThat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1912752463
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobDeleteHandler.java:
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_CONFLICT;
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.DIRECTORY_NOT_EMPTY_DELETE;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for delete over Blob endpoint. Blob endpoint for flat-namespace
+ * account does not support director delete. This class is responsible for
Review Comment:
typo: directory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916387329
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
Review Comment:
should have check for dstPathStatus != null as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917777595
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
Review Comment:
This will go once Ingress handler changes are taken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917838485
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java:
##########
@@ -201,4 +238,25 @@ public int getAcquireRetryCount() {
public TracingContext getTracingContext() {
return tracingContext;
}
+
Review Comment:
Added java doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937021494
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,634 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.TimeoutException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COLON;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = getPathInformation(src, tracingContext);
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(
+ src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * create renameJson file.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as rename on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(),
+ false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(COLON);
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path, getDstPathForBlob(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus != null && dstPathStatus.getResult() != null
+ && (srcCopyPath.equals(getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ final long maxWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressMaxWaitMillis();
+ long startTime = System.currentTimeMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
+ == BlobCopyProgress.PENDING) {
+ if (System.currentTimeMillis() - startTime > maxWait) {
+ throw new TimeoutException(
+ String.format("Blob copy progress wait time exceeded "
Review Comment:
As discussed offline, this is not needed.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/UriUtils.java:
##########
@@ -230,6 +240,28 @@ private static String replacedUrl(String baseUrl, String oldString, String newSt
return sb.toString();
}
+ public static boolean isKeyForDirectorySet(String key, Set<String> dirSet) {
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2626886677
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 13 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 5s | | trunk passed |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 46s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| -1 :x: | javadoc | 0m 27s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/21/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 25s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/21/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 117m 57s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/21/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux dee7206665fa 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dc6cf6fd37bb056329d2be7635080342d52ebf40 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/21/testReport/ |
| Max. process+thread count | 562 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/21/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2629084990
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 17 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 57s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 32s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/29/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 23 unchanged - 3 fixed = 24 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 25s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 47s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 130m 48s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/29/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 46fb527fcc90 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 621d6901795b5be1f85e36e6249fa5428cd9ee67 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/29/testReport/ |
| Max. process+thread count | 531 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/29/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931756998
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -338,6 +353,7 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
requestHeaders);
op.execute(tracingContext);
+ fixAtomicEntriesInListResults(op, tracingContext);
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2629300545
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 17m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 17 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 23s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 21s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 24s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 148m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/30/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 821423c8a4a4 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fff7ffcc9e8952287ff088d4b136fe738b3a2c6b |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/30/testReport/ |
| Max. process+thread count | 565 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/30/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2630520767
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 20 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 33s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 2s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/31/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 6 new + 23 unchanged - 3 fixed = 29 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 28s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/31/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 5 new + 10 unchanged - 1 fixed = 15 total (was 11) |
| -1 :x: | javadoc | 0m 25s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/31/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 5 new + 10 unchanged - 1 fixed = 15 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 23s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 34s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 130m 32s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/31/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 9226e7503c33 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5765c5a5da2411b21e2e1322efcf984cccc4e9e6 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/31/testReport/ |
| Max. process+thread count | 585 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/31/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1918225101
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -359,6 +375,34 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
return op;
}
+ private void fixAtomicEntriesInListResults(final AbfsRestOperation op,
+ final TracingContext tracingContext) throws AzureBlobFileSystemException {
+ /*
+ * Crashed HBase log rename recovery is done by Filesystem.getFileStatus and
+ * Filesystem.listStatus.
+ */
+ if (tracingContext == null
+ || tracingContext.getOpType() != FSOperationType.LISTSTATUS
+ || op == null || op.getResult() == null
+ || op.getResult().getStatusCode() != HTTP_OK) {
+ return;
+ }
+ BlobListResultSchema listResultSchema
+ = (BlobListResultSchema) op.getResult().getListResultSchema();
+ if (listResultSchema == null) {
+ return;
+ }
+ List<BlobListResultEntrySchema> filteredEntries = new ArrayList<>();
+ for (BlobListResultEntrySchema entry : listResultSchema.paths()) {
+ if (!takeListPathAtomicRenameKeyAction(entry.path(),
+ (int) (long) entry.contentLength(), tracingContext)) {
Review Comment:
Taken!
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -499,21 +556,50 @@ public AbfsRestOperation breakLease(final String path,
* @param isMetadataIncompleteState was there a rename failure due to
* incomplete metadata state?
* @param isNamespaceEnabled whether namespace enabled account or not
- * @return result of rename operation
- * @throws IOException if rename operation fails.
+ *
+ * @return AbfsClientRenameResult result of rename operation indicating the
+ * AbfsRest operation, rename recovery and incomplete metadata state failure.
+ *
+ * @throws IOException failure, excluding any recovery from overload failures.
*/
@Override
public AbfsClientRenameResult renamePath(final String source,
- final String destination,
- final String continuation,
- final TracingContext tracingContext,
- final String sourceEtag,
- final boolean isMetadataIncompleteState,
- final boolean isNamespaceEnabled) throws IOException {
- /**
- * TODO: [FnsOverBlob] To be implemented as part of rename-delete over blob endpoint work. <a href="https://issues.apache.org/jira/browse/HADOOP-19233">HADOOP-19233</a>.
- */
- throw new NotImplementedException("Rename operation on Blob endpoint yet to be implemented.");
+ final String destination,
+ final String continuation,
+ final TracingContext tracingContext,
+ String sourceEtag,
+ boolean isMetadataIncompleteState,
+ boolean isNamespaceEnabled)
+ throws IOException {
+ BlobRenameHandler blobRenameHandler = getBlobRenameHandler(source,
+ destination, sourceEtag, isAtomicRenameKey(source), tracingContext
+ );
+ incrementAbfsRenamePath();
+ if (blobRenameHandler.execute()) {
+ final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
+ final URL url = createRequestUrl(destination, abfsUriQueryBuilder.toString());
+ final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
+ final AbfsRestOperation successOp = getAbfsRestOperation(
+ AbfsRestOperationType.RenamePath, HTTP_METHOD_PUT,
+ url, requestHeaders);
+ successOp.hardSetResult(HttpURLConnection.HTTP_OK);
+ return new AbfsClientRenameResult(successOp, true, false);
+ } else {
+ throw new AbfsRestOperationException(HTTP_INTERNAL_ERROR,
+ AzureServiceErrorCode.UNKNOWN.getErrorCode(),
+ "FNS-Blob Rename was not successfull",
Review Comment:
Taken!
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobDeleteHandler.java:
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_CONFLICT;
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.DIRECTORY_NOT_EMPTY_DELETE;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for delete over Blob endpoint. Blob endpoint for flat-namespace
+ * account does not support director delete. This class is responsible for
Review Comment:
Resolved!
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
Review Comment:
Taken!
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1318,6 +1421,128 @@ public static String getDirectoryQueryParameter(final String path) {
return directory;
}
+ public boolean isAtomicRenameKey(String key) {
+ return isKeyForDirectorySet(key, azureAtomicRenameDirSet);
+ }
+
+ /**
+ * Action to be taken when atomic-key is present on a getPathStatus path.
+ *
+ * @param path path of the pendingJson for the atomic path.
+ * @param pathLease lease on the path.
+ * @param tracingContext tracing context.
+ *
+ * @throws AzureBlobFileSystemException server error or the path is renamePending json file and action is taken.
+ */
+ public void takeGetPathStatusAtomicRenameKeyAction(final Path path,
+ final AbfsLease pathLease,
+ final TracingContext tracingContext)
+ throws AzureBlobFileSystemException {
+ if (path == null || path.isRoot() || !isAtomicRenameKey(path.toUri().getPath())) {
+ return;
+ }
+ AbfsRestOperation pendingJsonFileStatus;
+ Path pendingJsonPath = new Path(path.getParent(),
+ path.toUri().getPath() + RenameAtomicity.SUFFIX);
+ try {
+ pendingJsonFileStatus = getPathStatus(
+ pendingJsonPath.toUri().getPath(), tracingContext, null, false);
+ if (checkIsDir(pendingJsonFileStatus.getResult())) {
+ return;
+ }
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_NOT_FOUND) {
+ return;
+ }
+ throw ex;
+ }
+
+ boolean renameSrcHasChanged;
+ try {
+ RenameAtomicity renameAtomicity = getRedoRenameAtomicity(
+ pendingJsonPath, Integer.parseInt(pendingJsonFileStatus.getResult()
+ .getResponseHeader(HttpHeaderConfigurations.CONTENT_LENGTH)),
+ tracingContext, pathLease);
+ renameAtomicity.redo();
+ renameSrcHasChanged = false;
+ } catch (AbfsRestOperationException ex) {
+ /*
+ * At this point, the source marked by the renamePending json file, might have
+ * already got renamed by some parallel thread, or at this point, the path
+ * would have got modified which would result in eTag change, which would lead
+ * to a HTTP_CONFLICT. In this case, no more operation needs to be taken, and
+ * the calling getPathStatus can return this source path as result.
+ */
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_NOT_FOUND
+ || ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ renameSrcHasChanged = true;
+ } else {
+ throw ex;
+ }
+ }
+ if (!renameSrcHasChanged) {
+ throw new AbfsRestOperationException(
+ AzureServiceErrorCode.PATH_NOT_FOUND.getStatusCode(),
+ AzureServiceErrorCode.PATH_NOT_FOUND.getErrorCode(),
+ ATOMIC_DIR_RENAME_RECOVERY_ON_GET_PATH_EXCEPTION,
+ null);
+ }
+ }
+
+ /**
+ * Action to be taken when atomic-key is present on a listPath path.
+ *
+ * @param path path of the pendingJson for the atomic path.
+ * @param renamePendingJsonLen length of the pendingJson file.
+ * @param tracingContext tracing context.
+ *
+ * @return true if action is taken.
+ * @throws AzureBlobFileSystemException server error
+ */
+ private boolean takeListPathAtomicRenameKeyAction(final Path path,
+ final int renamePendingJsonLen,
+ final TracingContext tracingContext)
+ throws AzureBlobFileSystemException {
+ if (path == null || path.isRoot() || !isAtomicRenameKey(
+ path.toUri().getPath()) || !path.toUri()
+ .getPath()
+ .endsWith(RenameAtomicity.SUFFIX)) {
+ return false;
+ }
+ try {
+ RenameAtomicity renameAtomicity
+ = getRedoRenameAtomicity(path, renamePendingJsonLen,
+ tracingContext, null);
+ renameAtomicity.redo();
+ } catch (AbfsRestOperationException ex) {
+ /*
+ * At this point, the source marked by the renamePending json file, might have
+ * already got renamed by some parallel thread, or at this point, the path
+ * would have got modified which would result in eTag change, which would lead
+ * to a HTTP_CONFLICT. In this case, no more operation needs to be taken, but
+ * since this is a renamePendingJson file and would be deleted by the redo operation,
+ * the calling listPath should not return this json path as result.
+ */
+ if (ex.getStatusCode() != HttpURLConnection.HTTP_NOT_FOUND
+ && ex.getStatusCode() != HttpURLConnection.HTTP_CONFLICT) {
+ throw ex;
+ }
+ }
+ return true;
Review Comment:
In the above if condition we are filtering out paths for which we don't have to do the rename redo and returning false there itself. In case where path require rename redo, we are returning true if no exception is raised.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobDeleteHandler.java:
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_CONFLICT;
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.DIRECTORY_NOT_EMPTY_DELETE;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for delete over Blob endpoint. Blob endpoint for flat-namespace
+ * account does not support director delete. This class is responsible for
+ * deleting the blobs and creating the parent directory marker file if needed.
+ */
+public class BlobDeleteHandler extends ListActionTaker {
+
+ private final Path path;
+
+ private final boolean recursive;
+
+ private boolean nonRecursiveDeleteDirectoryFailed = false;
+
+ private final TracingContext tracingContext;
+
+ private final AtomicInteger deleteCount = new AtomicInteger(0);
+
+ /** Constructor
+ *
+ * @param path path to delete.
+ * @param recursive if true, delete the path recursively.
+ * @param abfsBlobClient client to use for blob operations.
+ * @param tracingContext tracing context.
+ */
+ public BlobDeleteHandler(final Path path,
+ final boolean recursive,
+ final AbfsBlobClient abfsBlobClient,
+ final TracingContext tracingContext) {
+ super(path, abfsBlobClient, tracingContext);
+ this.path = path;
+ this.recursive = recursive;
+ this.tracingContext = tracingContext;
+ }
+
+ /**{@inheritDoc}
+ *
+ * @return the maximum number of parallelism for delete operation.
+ */
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobDeleteDirConsumptionParallelism();
+ }
+
+ /** Delete the path.
+ *
+ * @param path path to delete.
+ * @return true if the path is deleted.
+ * @throws AzureBlobFileSystemException server error.
+ */
+ private boolean deleteInternal(final Path path)
+ throws AzureBlobFileSystemException {
+ getAbfsClient().deleteBlobPath(path, null, tracingContext);
+ deleteCount.incrementAndGet();
+ return true;
+ }
+
+ /**
+ * Orchestrate the delete operation.
+ *
+ * @return true if the delete operation is successful.
+ * @throws AzureBlobFileSystemException if deletion fails due to server error or path doesn't exist.
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ /*
+ * ABFS is not aware if it's a file or directory. So, we need to list the
+ * path and delete the listed objects. The listing returns the children of
+ * the path and not the path itself.
+ */
+ listRecursiveAndTakeAction();
+ if (nonRecursiveDeleteDirectoryFailed) {
+ throw new AbfsRestOperationException(HTTP_CONFLICT,
+ DIRECTORY_NOT_EMPTY_DELETE.getErrorCode(),
+ DIRECTORY_NOT_EMPTY_DELETE.getErrorMessage(),
+ new PathIOException(path.toString(),
+ "Non-recursive delete of non-empty directory"));
+ }
+ tracingContext.setOperatedBlobCount(deleteCount.get() + 1);
+ /*
+ * If path is actually deleted.
+ */
+ boolean deleted;
+ try {
+ /*
+ * Delete the required path.
+ * Directory needs to be safely delete the path, as the path can be implicit.
Review Comment:
Updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1918217050
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java:
##########
@@ -227,6 +236,13 @@ public void testDeleteIdempotency() throws Exception {
public void testDeleteIdempotencyTriggerHttp404() throws Exception {
final AzureBlobFileSystem fs = getFileSystem();
+ /*
+ * Delete call for a folder on DFS endpoint is one-server call and the
+ * orchestration of delete of paths inside the directory. For Blob
+ * endpoint, the orchestration would be done by the client. The idempotency
+ * issue would not happen for blob endpoint.
+ */
+ assertTrue(fs.getAbfsClient() instanceof AbfsDfsClient);
Review Comment:
can use getAbfsServiceType()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1919828723
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -192,10 +239,1304 @@ public void testRenameWithNoDestinationParentDir() throws Exception {
// Verify that metadata was in an incomplete state after the rename
// failure, and we retired the rename once more.
IOStatistics ioStatistics = fs.getIOStatistics();
+ AbfsClient client = fs.getAbfsStore().getClient();
IOStatisticAssertions.assertThatStatisticCounter(ioStatistics,
RENAME_PATH_ATTEMPTS.getStatName())
- .describedAs("There should be 2 rename attempts if metadata "
- + "incomplete state failure is hit")
- .isEqualTo(2);
+ .describedAs("For Dfs endpoint: There should be 2 rename "
+ + "attempts if metadata incomplete state failure is hit."
+ + "For Blob endpoint: There would be only one rename attempt which "
+ + "would have a failed precheck.")
+ .isEqualTo(client instanceof AbfsDfsClient ? 2 : 1);
+ }
+
+ @Test
+ public void testRenameToRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ fs.mkdirs(new Path("/src1/src2"));
+ assertTrue(fs.rename(new Path("/src1/src2"), new Path("/")));
+ assertTrue(fs.exists(new Path("/src2")));
+ }
+
+ @Test
+ public void testRenameNotFoundBlobToEmptyRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assertFalse(fs.rename(new Path("/file"), new Path("/")));
+ }
+
+ private void assumeNonHnsAccountBlobEndpoint(final AzureBlobFileSystem fs) {
+ assertTrue(fs.getAbfsStore().getClient() instanceof AbfsBlobClient);
+ }
+
+ @Test(expected = IOException.class)
+ public void testRenameBlobToDstWithColonInPath() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.create(new Path("/src"));
+ fs.rename(new Path("/src"), new Path("/dst:file"));
+ }
+
+ @Test
+ public void testRenameBlobInSameDirectoryWithNoMarker() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) fs.getAbfsStore().getClient();
+ fs.create(new Path("/srcDir/dir/file"));
+ client.deleteBlobPath(new Path("/srcDir/dir"), null,
+ getTestTracingContext(fs, true));
+ assertTrue(fs.rename(new Path("/srcDir/dir"), new Path("/srcDir")));
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWhereDirectoryAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.mkdirs(new Path("testDir2/test4/test3"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3")));
+ if (getIsNamespaceEnabled(fs)
+ || fs.getAbfsClient() instanceof AbfsBlobClient) {
+ assertFalse(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ } else {
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ }
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWherePartAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.create(new Path("testDir2/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("testDir2/test4/"));
+ fs.create(new Path("testDir2/test4/file1"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ Assert.assertTrue(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3")));
+ assertFalse(fs.exists(new Path("testDir2/test4/file")));
+ assertTrue(fs.exists(new Path("testDir2/test4/file1")));
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file1")));
+ assertTrue(fs.exists(new Path("testDir2/test4/file1")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ }
+
+ /**
+ * Test that after completing rename for a directory which is enabled for
+ * AtomicRename, the RenamePending JSON file is deleted.
+ */
+ @Test
+ public void testRenamePendingJsonIsRemovedPostSuccessfulRename()
+ throws Exception {
+ final AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) addSpyHooksOnClient(fs);
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.setWorkingDirectory(new Path("/"));
+ fs.mkdirs(new Path("hbase/test1/test2/test3"));
+ fs.create(new Path("hbase/test1/test2/test3/file"));
+ fs.create(new Path("hbase/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("hbase/test4/"));
+ fs.create(new Path("hbase/test4/file1"));
+ final Integer[] correctDeletePathCount = new Integer[1];
+ correctDeletePathCount[0] = 0;
+ Mockito.doAnswer(answer -> {
+ final String correctDeletePath = "/hbase/test1/test2/test3" + SUFFIX;
+ if (correctDeletePath.equals(
+ ((Path) answer.getArgument(0)).toUri().getPath())) {
+ correctDeletePathCount[0] = 1;
+ }
+ return null;
+ })
+ .when(client)
+ .deleteBlobPath(Mockito.any(Path.class), Mockito.nullable(String.class),
+ Mockito.any(TracingContext.class));
+ assertTrue(fs.rename(new Path("hbase/test1/test2/test3"),
+ new Path("hbase/test4")));
+ assertEquals("RenamePendingJson should be deleted",
+ 1,
+ (int) correctDeletePathCount[0]);
+ }
+
+ private AbfsClient addSpyHooksOnClient(final AzureBlobFileSystem fs) {
+ AzureBlobFileSystemStore store = Mockito.spy(fs.getAbfsStore());
+ Mockito.doReturn(store).when(fs).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+ return client;
+ }
+
+ /**
+ * Test for a directory in /hbase directory. To simulate the crash of process,
+ * test will throw an exception with 403 on a copy of one of the blob.<br>
+ * ListStatus API will be called on the directory. Expectation is that the ListStatus
+ * API of {@link AzureBlobFileSystem} should recover the paused rename.
+ */
+ @Test
+ public void testHBaseHandlingForFailedRenameWithListRecovery()
Review Comment:
This and the test below are almost similar, code can be reused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936767113
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -341,7 +360,7 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
abfsUriQueryBuilder.addQuery(QUERY_PARAM_DELIMITER, FORWARD_SLASH);
}
abfsUriQueryBuilder.addQuery(QUERY_PARAM_MAX_RESULTS, String.valueOf(listMaxResults));
- appendSASTokenToQuery(relativePath, SASTokenProvider.FIXED_SAS_STORE_OPERATION, abfsUriQueryBuilder);
+ appendSASTokenToQuery(relativePath, SASTokenProvider.LIST_OPERATION, abfsUriQueryBuilder);
Review Comment:
Taken!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936107469
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -220,7 +219,7 @@ public void initialize(URI uri, Configuration configuration)
this.setWorkingDirectory(this.getHomeDirectory());
TracingContext initFSTracingContext = new TracingContext(clientCorrelationId,
- fileSystemId, FSOperationType.INIT, tracingHeaderFormat, listener);
+ fileSystemId, FSOperationType.INIT, tracingHeaderFormat, listener);
Review Comment:
undo unrelated changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2579979013
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 58s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 13 unchanged - 3 fixed = 13 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 25s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 39s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 134m 12s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux fc9e167f5b5d 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b4f157e7ba16b7bdcff814c37db2bb97403712c4 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/8/testReport/ |
| Max. process+thread count | 550 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1910302591
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -359,6 +375,34 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
return op;
}
+ private void fixAtomicEntriesInListResults(final AbfsRestOperation op,
+ final TracingContext tracingContext) throws AzureBlobFileSystemException {
+ /*
+ * Crashed HBase log rename recovery is done by Filesystem.getFileStatus and
+ * Filesystem.listStatus.
+ */
+ if (tracingContext == null
+ || tracingContext.getOpType() != FSOperationType.LISTSTATUS
+ || op == null || op.getResult() == null
+ || op.getResult().getStatusCode() != HTTP_OK) {
+ return;
+ }
+ BlobListResultSchema listResultSchema
+ = (BlobListResultSchema) op.getResult().getListResultSchema();
+ if (listResultSchema == null) {
+ return;
+ }
+ List<BlobListResultEntrySchema> filteredEntries = new ArrayList<>();
+ for (BlobListResultEntrySchema entry : listResultSchema.paths()) {
+ if (!takeListPathAtomicRenameKeyAction(entry.path(),
+ (int) (long) entry.contentLength(), tracingContext)) {
Review Comment:
double casting is not needed, we can use something like :-
Long longValue = 12345L;
int intValue = longValue.intValue();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1918218593
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java:
##########
@@ -314,4 +347,278 @@ public void deleteBlobDirParallelThreadToDeleteOnDifferentTracingContext()
fs.delete(new Path("/testDir"), true);
fs.close();
}
+
+ private void assumeBlobClient() throws IOException {
Review Comment:
[optional] This method can be avoided and all calls to it can be replaced by getAbfsServiceType() == AbfsServiceType.BLOB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1920024724
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -192,10 +239,1304 @@ public void testRenameWithNoDestinationParentDir() throws Exception {
// Verify that metadata was in an incomplete state after the rename
// failure, and we retired the rename once more.
IOStatistics ioStatistics = fs.getIOStatistics();
+ AbfsClient client = fs.getAbfsStore().getClient();
IOStatisticAssertions.assertThatStatisticCounter(ioStatistics,
RENAME_PATH_ATTEMPTS.getStatName())
- .describedAs("There should be 2 rename attempts if metadata "
- + "incomplete state failure is hit")
- .isEqualTo(2);
+ .describedAs("For Dfs endpoint: There should be 2 rename "
+ + "attempts if metadata incomplete state failure is hit."
+ + "For Blob endpoint: There would be only one rename attempt which "
+ + "would have a failed precheck.")
+ .isEqualTo(client instanceof AbfsDfsClient ? 2 : 1);
+ }
+
+ @Test
+ public void testRenameToRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ fs.mkdirs(new Path("/src1/src2"));
+ assertTrue(fs.rename(new Path("/src1/src2"), new Path("/")));
+ assertTrue(fs.exists(new Path("/src2")));
+ }
+
+ @Test
+ public void testRenameNotFoundBlobToEmptyRoot() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assertFalse(fs.rename(new Path("/file"), new Path("/")));
+ }
+
+ private void assumeNonHnsAccountBlobEndpoint(final AzureBlobFileSystem fs) {
+ assertTrue(fs.getAbfsStore().getClient() instanceof AbfsBlobClient);
+ }
+
+ @Test(expected = IOException.class)
+ public void testRenameBlobToDstWithColonInPath() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.create(new Path("/src"));
+ fs.rename(new Path("/src"), new Path("/dst:file"));
+ }
+
+ @Test
+ public void testRenameBlobInSameDirectoryWithNoMarker() throws Exception {
+ AzureBlobFileSystem fs = getFileSystem();
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) fs.getAbfsStore().getClient();
+ fs.create(new Path("/srcDir/dir/file"));
+ client.deleteBlobPath(new Path("/srcDir/dir"), null,
+ getTestTracingContext(fs, true));
+ assertTrue(fs.rename(new Path("/srcDir/dir"), new Path("/srcDir")));
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWhereDirectoryAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.mkdirs(new Path("testDir2/test4/test3"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3")));
+ if (getIsNamespaceEnabled(fs)
+ || fs.getAbfsClient() instanceof AbfsBlobClient) {
+ assertFalse(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ } else {
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ }
+ }
+
+ /**
+ * <pre>
+ * Test to check behaviour of rename API if the destination directory is already
+ * there. The HNS call and the one for Blob endpoint should have same behaviour.
+ *
+ * /testDir2/test1/test2/test3 contains (/file)
+ * There is another path that exists: /testDir2/test4/test3
+ * On rename(/testDir2/test1/test2/test3, /testDir2/test4).
+ * </pre>
+ *
+ * Expectation for HNS / Blob endpoint:<ol>
+ * <li>Rename should fail</li>
+ * <li>No file should be transferred to destination directory</li>
+ * </ol>
+ */
+ @Test
+ public void testPosixRenameDirectoryWherePartAlreadyThereOnDestination()
+ throws Exception {
+ final AzureBlobFileSystem fs = this.getFileSystem();
+ fs.mkdirs(new Path("testDir2/test1/test2/test3"));
+ fs.create(new Path("testDir2/test1/test2/test3/file"));
+ fs.create(new Path("testDir2/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("testDir2/test4/"));
+ fs.create(new Path("testDir2/test4/file1"));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ Assert.assertTrue(fs.rename(new Path("testDir2/test1/test2/test3"),
+ new Path("testDir2/test4")));
+ assertTrue(fs.exists(new Path("testDir2")));
+ assertTrue(fs.exists(new Path("testDir2/test1/test2")));
+ assertTrue(fs.exists(new Path("testDir2/test4")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3")));
+ assertFalse(fs.exists(new Path("testDir2/test4/file")));
+ assertTrue(fs.exists(new Path("testDir2/test4/file1")));
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file")));
+ assertTrue(fs.exists(new Path("testDir2/test4/test3/file1")));
+ assertTrue(fs.exists(new Path("testDir2/test4/file1")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file")));
+ assertFalse(fs.exists(new Path("testDir2/test1/test2/test3/file1")));
+ }
+
+ /**
+ * Test that after completing rename for a directory which is enabled for
+ * AtomicRename, the RenamePending JSON file is deleted.
+ */
+ @Test
+ public void testRenamePendingJsonIsRemovedPostSuccessfulRename()
+ throws Exception {
+ final AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) addSpyHooksOnClient(fs);
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.setWorkingDirectory(new Path("/"));
+ fs.mkdirs(new Path("hbase/test1/test2/test3"));
+ fs.create(new Path("hbase/test1/test2/test3/file"));
+ fs.create(new Path("hbase/test1/test2/test3/file1"));
+ fs.mkdirs(new Path("hbase/test4/"));
+ fs.create(new Path("hbase/test4/file1"));
+ final Integer[] correctDeletePathCount = new Integer[1];
+ correctDeletePathCount[0] = 0;
+ Mockito.doAnswer(answer -> {
+ final String correctDeletePath = "/hbase/test1/test2/test3" + SUFFIX;
+ if (correctDeletePath.equals(
+ ((Path) answer.getArgument(0)).toUri().getPath())) {
+ correctDeletePathCount[0] = 1;
+ }
+ return null;
+ })
+ .when(client)
+ .deleteBlobPath(Mockito.any(Path.class), Mockito.nullable(String.class),
+ Mockito.any(TracingContext.class));
+ assertTrue(fs.rename(new Path("hbase/test1/test2/test3"),
+ new Path("hbase/test4")));
+ assertEquals("RenamePendingJson should be deleted",
+ 1,
+ (int) correctDeletePathCount[0]);
+ }
+
+ private AbfsClient addSpyHooksOnClient(final AzureBlobFileSystem fs) {
+ AzureBlobFileSystemStore store = Mockito.spy(fs.getAbfsStore());
+ Mockito.doReturn(store).when(fs).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+ return client;
+ }
+
+ /**
+ * Test for a directory in /hbase directory. To simulate the crash of process,
+ * test will throw an exception with 403 on a copy of one of the blob.<br>
+ * ListStatus API will be called on the directory. Expectation is that the ListStatus
+ * API of {@link AzureBlobFileSystem} should recover the paused rename.
+ */
+ @Test
+ public void testHBaseHandlingForFailedRenameWithListRecovery()
+ throws Exception {
+ AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) addSpyHooksOnClient(fs);
+ String srcPath = "hbase/test1/test2";
+ final String failedCopyPath = srcPath + "/test3/file1";
+ fs.setWorkingDirectory(new Path("/"));
+ fs.mkdirs(new Path(srcPath));
+ fs.mkdirs(new Path(srcPath, "test3"));
+ fs.create(new Path(srcPath + "/test3/file"));
+ fs.create(new Path(failedCopyPath));
+ fs.mkdirs(new Path("hbase/test4/"));
+ fs.create(new Path("hbase/test4/file1"));
+ crashRenameAndRecover(fs, client, srcPath, (abfsFs) -> {
+ abfsFs.listStatus(new Path(srcPath).getParent());
+ return null;
+ });
+ }
+
+ /**
+ * Test for a directory in /hbase directory. To simulate the crash of process,
+ * test will throw an exception with 403 on a copy of one of the blob. The
+ * source directory is a nested directory.<br>
+ * GetFileStatus API will be called on the directory. Expectation is that the
+ * GetFileStatus API of {@link AzureBlobFileSystem} should recover the paused
+ * rename.
+ */
+ @Test
+ public void testHBaseHandlingForFailedRenameWithGetFileStatusRecovery()
+ throws Exception {
+ AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ AbfsBlobClient client = (AbfsBlobClient) addSpyHooksOnClient(fs);
+ String srcPath = "hbase/test1/test2";
+ final String failedCopyPath = srcPath + "/test3/file1";
+ fs.setWorkingDirectory(new Path("/"));
+ fs.mkdirs(new Path(srcPath));
+ fs.mkdirs(new Path(srcPath, "test3"));
+ fs.create(new Path(srcPath + "/test3/file"));
+ fs.create(new Path(failedCopyPath));
+ fs.mkdirs(new Path("hbase/test4/"));
+ fs.create(new Path("hbase/test4/file1"));
+ crashRenameAndRecover(fs, client, srcPath, (abfsFs) -> {
+ abfsFs.exists(new Path(srcPath));
+ return null;
+ });
+ }
+
+ private void crashRenameAndRecover(final AzureBlobFileSystem fs,
+ AbfsBlobClient client,
+ final String srcPath,
+ final FunctionRaisingIOE<AzureBlobFileSystem, Void> recoveryCallable)
+ throws Exception {
+ crashRename(fs, client, srcPath);
+ AzureBlobFileSystem fs2 = Mockito.spy(getFileSystem());
+ fs2.setWorkingDirectory(new Path(ROOT_PATH));
+ client = (AbfsBlobClient) addSpyHooksOnClient(fs2);
+ int[] renameJsonDeleteCounter = new int[1];
+ Mockito.doAnswer(answer -> {
+ if ((ROOT_PATH + srcPath + SUFFIX)
+ .equalsIgnoreCase(((Path) answer.getArgument(0)).toUri().getPath())) {
+ renameJsonDeleteCounter[0] = 1;
+ }
+ return answer.callRealMethod();
+ })
+ .when(client)
+ .deleteBlobPath(Mockito.any(Path.class), Mockito.nullable(String.class),
+ Mockito.any(TracingContext.class));
+ recoveryCallable.apply(fs2);
+ Assertions.assertThat(renameJsonDeleteCounter[0])
+ .describedAs("RenamePendingJson should be deleted")
+ .isEqualTo(1);
+ //List would complete the rename orchestration.
+ assertFalse(fs2.exists(new Path("hbase/test1/test2")));
+ assertFalse(fs2.exists(new Path("hbase/test1/test2/test3")));
+ assertTrue(fs2.exists(new Path("hbase/test4/test2/test3")));
+ assertFalse(fs2.exists(new Path("hbase/test1/test2/test3/file")));
+ assertTrue(fs2.exists(new Path("hbase/test4/test2/test3/file")));
+ assertFalse(fs2.exists(new Path("hbase/test1/test2/test3/file1")));
+ assertTrue(fs2.exists(new Path("hbase/test4/test2/test3/file1")));
+ }
+
+ private void crashRename(final AzureBlobFileSystem fs,
+ final AbfsBlobClient client,
+ final String srcPath) throws Exception {
+ BlobRenameHandler[] blobRenameHandlers = new BlobRenameHandler[1];
+ AbfsClientTestUtil.mockGetRenameBlobHandler(client,
+ blobRenameHandler -> {
+ blobRenameHandlers[0] = blobRenameHandler;
+ return null;
+ });
+ //Fail rename orchestration on path hbase/test1/test2/test3/file1
+ Mockito.doThrow(new AbfsRestOperationException(HTTP_FORBIDDEN, "", "",
+ new Exception()))
+ .when(client)
+ .copyBlob(Mockito.any(Path.class), Mockito.any(Path.class),
+ Mockito.nullable(String.class),
+ Mockito.any(TracingContext.class));
+ LambdaTestUtils.intercept(AccessDeniedException.class, () -> {
+ fs.rename(new Path(srcPath),
+ new Path("hbase/test4"));
+ });
+ //Release all the leases taken by atomic rename orchestration
+ List<AbfsLease> leases = new ArrayList<>(blobRenameHandlers[0].getLeases());
+ for (AbfsLease lease : leases) {
+ lease.free();
+ }
+ }
+
+ /**
+ * Simulates a scenario where HMaster in Hbase starts up and executes listStatus
+ * API on the directory that has to be renamed by some other executor-machine.
+ * The scenario is that RenamePending JSON is created but before it could be
+ * appended, it has been opened by the HMaster. The HMaster will delete it. The
+ * machine doing rename would have to recreate the JSON file.
+ * ref: <a href="https://issues.apache.org/jira/browse/HADOOP-12678">issue</a>
+ */
+ @Test
+ public void testHbaseListStatusBeforeRenamePendingFileAppendedWithIngressOnBlob()
+ throws Exception {
+ final AzureBlobFileSystem fs = Mockito.spy(this.getFileSystem());
+ assumeNonHnsAccountBlobEndpoint(fs);
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ testRenamePreRenameFailureResolution(fs);
+ testAtomicityRedoInvalidFile(fs);
+ }
+
+ private void testRenamePreRenameFailureResolution(final AzureBlobFileSystem fs)
Review Comment:
comments for tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1931699236
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/BlobRenameHandler.java:
##########
@@ -0,0 +1,652 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode;
+import org.apache.hadoop.fs.azurebfs.enums.BlobCopyProgress;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_FAILED;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COPY_STATUS_SUCCESS;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ROOT_PATH;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_ID;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_SOURCE;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS;
+import static org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations.X_MS_COPY_STATUS_DESCRIPTION;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_ABORTED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.COPY_BLOB_FAILED;
+import static org.apache.hadoop.fs.azurebfs.contracts.services.AzureServiceErrorCode.RENAME_DESTINATION_PARENT_PATH_NOT_FOUND;
+
+/**
+ * Orchestrator for rename over Blob endpoint. Handles both directory and file
+ * renames. Blob Endpoint does not expose rename API, this class is responsible
+ * for copying the blobs and deleting the source blobs.
+ * <p>
+ * For directory rename, it recursively lists the blobs in the source directory and
+ * copies them to the destination directory.
+ */
+public class BlobRenameHandler extends ListActionTaker {
+
+ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
+
+ private final String srcEtag;
+
+ private final Path src, dst;
+
+ private final boolean isAtomicRename, isAtomicRenameRecovery;
+
+ private final TracingContext tracingContext;
+
+ private AbfsLease srcAbfsLease;
+
+ private String srcLeaseId;
+
+ private final List<AbfsLease> leases = new ArrayList<>();
+
+ private final AtomicInteger operatedBlobCount = new AtomicInteger(0);
+
+ /** Constructor.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param abfsClient AbfsBlobClient to use for the rename operation
+ * @param srcEtag eTag of the source path
+ * @param isAtomicRename true if the rename operation is atomic
+ * @param isAtomicRenameRecovery true if the rename operation is a recovery of a previous failed atomic rename operation
+ * @param tracingContext object of tracingContext used for the tracing of the server calls.
+ */
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final TracingContext tracingContext) {
+ super(new Path(src), abfsClient, tracingContext);
+ this.srcEtag = srcEtag;
+ this.tracingContext = tracingContext;
+ this.src = new Path(src);
+ this.dst = new Path(dst);
+ this.isAtomicRename = isAtomicRename;
+ this.isAtomicRenameRecovery = isAtomicRenameRecovery;
+ }
+
+ public BlobRenameHandler(final String src,
+ final String dst,
+ final AbfsBlobClient abfsClient,
+ final String srcEtag,
+ final boolean isAtomicRename,
+ final boolean isAtomicRenameRecovery,
+ final AbfsLease srcAbfsLease,
+ final TracingContext tracingContext) {
+ this(src, dst, abfsClient, srcEtag, isAtomicRename, isAtomicRenameRecovery,
+ tracingContext);
+ this.srcAbfsLease = srcAbfsLease;
+ }
+
+ @Override
+ int getMaxConsumptionParallelism() {
+ return getAbfsClient().getAbfsConfiguration()
+ .getBlobRenameDirConsumptionParallelism();
+ }
+
+ /**
+ * Orchestrates the rename operation.
+ *
+ * @return AbfsClientRenameResult containing the result of the rename operation
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ public boolean execute() throws AzureBlobFileSystemException {
+ PathInformation pathInformation = new PathInformation();
+ boolean result = false;
+ if (preCheck(src, dst, pathInformation)) {
+ RenameAtomicity renameAtomicity = null;
+ if (pathInformation.getIsDirectory()
+ && pathInformation.getIsImplicit()) {
+ AbfsRestOperation createMarkerOp = getAbfsClient().createPath(src.toUri().getPath(),
+ false, false, null,
+ false, null, null, tracingContext);
+ pathInformation.setETag(extractEtagHeader(createMarkerOp.getResult()));
+ }
+ try {
+ if (isAtomicRename) {
+ /*
+ * Conditionally get a lease on the source blob to prevent other writers
+ * from changing it. This is used for correctness in HBase when log files
+ * are renamed. When the HBase master renames a log file folder, the lease
+ * locks out other writers. This prevents a region server that the master
+ * thinks is dead, but is still alive, from committing additional updates.
+ * This is different than when HBase runs on HDFS, where the region server
+ * recovers the lease on a log file, to gain exclusive access to it, before
+ * it splits it.
+ */
+ if (srcAbfsLease == null) {
+ srcAbfsLease = takeLease(src, srcEtag);
+ }
+ srcLeaseId = srcAbfsLease.getLeaseID();
+ if (!isAtomicRenameRecovery && pathInformation.getIsDirectory()) {
+ /*
+ * if it is not a resume of a previous failed atomic rename operation,
+ * perform the pre-rename operation.
+ */
+ renameAtomicity = getRenameAtomicity(pathInformation);
+ renameAtomicity.preRename();
+ }
+ }
+ if (pathInformation.getIsDirectory()) {
+ result = listRecursiveAndTakeAction() && finalSrcRename();
+ } else {
+ result = renameInternal(src, dst);
+ }
+ } finally {
+ if (srcAbfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (result) {
+ srcAbfsLease.cancelTimer();
+ } else {
+ srcAbfsLease.free();
+ }
+ }
+ }
+ if (result && renameAtomicity != null) {
+ renameAtomicity.postRename();
+ }
+ }
+ return result;
+ }
+
+ /** Final rename operation after all the blobs have been copied.
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean finalSrcRename() throws AzureBlobFileSystemException {
+ tracingContext.setOperatedBlobCount(operatedBlobCount.get() + 1);
+ try {
+ return renameInternal(src, dst);
+ } finally {
+ tracingContext.setOperatedBlobCount(null);
+ }
+ }
+
+ /** Gets the rename atomicity object.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @return RenameAtomicity object
+ */
+ @VisibleForTesting
+ public RenameAtomicity getRenameAtomicity(final PathInformation pathInformation) {
+ return new RenameAtomicity(src,
+ dst,
+ new Path(src.getParent(), src.getName() + RenameAtomicity.SUFFIX),
+ tracingContext,
+ pathInformation.getETag(),
+ getAbfsClient());
+ }
+
+ /** Takes a lease on the path.
+ *
+ * @param path path on which the lease is to be taken
+ * @param eTag eTag of the path
+ *
+ * @return object containing the lease information
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private AbfsLease takeLease(final Path path, final String eTag)
+ throws AzureBlobFileSystemException {
+ AbfsLease lease = new AbfsLease(getAbfsClient(), path.toUri().getPath(), false,
+ getAbfsClient().getAbfsConfiguration()
+ .getAtomicRenameLeaseRefreshDuration(),
+ eTag, tracingContext);
+ leases.add(lease);
+ return lease;
+ }
+
+ /** Checks if the path contains a colon.
+ *
+ * @param p path to check
+ *
+ * @return true if the path contains a colon
+ */
+ private boolean containsColon(Path p) {
+ return p.toUri().getPath().contains(":");
+ }
+
+ /**
+ * Since, server doesn't have a rename API and would not be able to check HDFS
+ * contracts, client would have to ensure that no HDFS contract is violated.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object in which path information of the source path would be stored
+ *
+ * @return true if the pre-checks pass
+ * @throws AzureBlobFileSystemException if server call fails or given paths are invalid.
+ */
+ private boolean preCheck(final Path src, final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ validateDestinationPath(src, dst);
+
+ setSrcPathInformation(src, pathInformation);
+ validateSourcePath(pathInformation);
+ validateDestinationPathNotExist(src, dst, pathInformation);
+ validateDestinationParentExist(src, dst, pathInformation);
+
+ return true;
+ }
+
+ /**
+ * Validate if the format of the destination path is correct and if the destination
+ * path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ *
+ * @throws AbfsRestOperationException if the destination path is invalid
+ */
+ private void validateDestinationPath(final Path src, final Path dst)
+ throws AbfsRestOperationException {
+ if (containsColon(dst)) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_BAD_REQUEST,
+ AzureServiceErrorCode.INVALID_RENAME_DESTINATION.getErrorCode(), null,
+ new PathIOException(dst.toUri().getPath(),
+ "Destination path contains colon"));
+ }
+
+ validateDestinationIsNotSubDir(src, dst);
+ }
+
+ /**
+ * Validate if the destination path is not a sub-directory of the source path.
+ *
+ * @param src source path
+ * @param dst destination path
+ */
+ private void validateDestinationIsNotSubDir(final Path src,
+ final Path dst) throws AbfsRestOperationException {
+ LOG.debug("Check if the destination is subDirectory");
+ Path nestedDstParent = dst.getParent();
+ if (nestedDstParent != null && nestedDstParent.toUri()
+ .getPath()
+ .indexOf(src.toUri().getPath()) == 0) {
+ LOG.info("Rename src: {} dst: {} failed as dst is subDir of src",
+ src, dst);
+ throw new AbfsRestOperationException(HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode(),
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorMessage(),
+ new Exception(
+ AzureServiceErrorCode.INVALID_RENAME_SOURCE_PATH.getErrorCode()));
+ }
+ }
+
+ /** Set the path information of the source path.
+ *
+ * @param src source path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void setSrcPathInformation(final Path src,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ pathInformation.copy(getPathInformation(src, tracingContext));
+ }
+
+ /**
+ * Validate if the source path exists and if the client knows the ETag of the source path,
+ * then the ETag should match with the server.
+ *
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the source path is not found or if the ETag of the source
+ * path does not match with the server.
+ */
+ private void validateSourcePath(final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ if (!pathInformation.getPathExists()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.SOURCE_PATH_NOT_FOUND.getErrorCode()));
+ }
+ if (srcEtag != null && !srcEtag.equals(pathInformation.getETag())) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ new Exception(
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode()));
+ }
+ }
+
+ /** Validate if the destination path does not exist.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the destination path already exists
+ */
+ private void validateDestinationPathNotExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ /*
+ * Destination path name can be same to that of source path name only in the
+ * case of a directory rename.
+ *
+ * In case the directory is being renamed to some other name, the destination
+ * check would happen on the AzureBlobFileSystem#rename method.
+ */
+ if (pathInformation.getIsDirectory() && dst.getName()
+ .equals(src.getName())) {
+ PathInformation dstPathInformation = getPathInformation(
+ dst,
+ tracingContext);
+ if (dstPathInformation.getPathExists()) {
+ LOG.info(
+ "Rename src: {} dst: {} failed as qualifiedDst already exists",
+ src, dst);
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_CONFLICT,
+ AzureServiceErrorCode.PATH_ALREADY_EXISTS.getErrorCode(), null,
+ null);
+ }
+ }
+ }
+
+ /** Validate if the parent of the destination path exists.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param pathInformation object containing the path information of the source path
+ *
+ * @throws AbfsRestOperationException if the parent of the destination path does not exist
+ */
+ private void validateDestinationParentExist(final Path src,
+ final Path dst,
+ final PathInformation pathInformation)
+ throws AzureBlobFileSystemException {
+ final Path nestedDstParent = dst.getParent();
+ if (!dst.isRoot() && nestedDstParent != null && !nestedDstParent.isRoot()
+ && (
+ !pathInformation.getIsDirectory() || !dst.getName()
+ .equals(src.getName()))) {
+ PathInformation nestedDstInfo = getPathInformation(
+ nestedDstParent,
+ tracingContext);
+ if (!nestedDstInfo.getPathExists() || !nestedDstInfo.getIsDirectory()) {
+ throw new AbfsRestOperationException(
+ HttpURLConnection.HTTP_NOT_FOUND,
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode(), null,
+ new Exception(
+ RENAME_DESTINATION_PARENT_PATH_NOT_FOUND.getErrorCode()));
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ boolean takeAction(final Path path) throws AzureBlobFileSystemException {
+ return renameInternal(path,
+ createDestinationPathForBlobPartOfRenameSrcDir(dst, path, src));
+ }
+
+ /** Renames the source path to the destination path.
+ *
+ * @param path source path
+ * @param destinationPathForBlobPartOfRenameSrcDir destination path
+ *
+ * @return true if rename is successful
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private boolean renameInternal(final Path path,
+ final Path destinationPathForBlobPartOfRenameSrcDir)
+ throws AzureBlobFileSystemException {
+ final String leaseId;
+ AbfsLease abfsLease = null;
+ if (isAtomicRename) {
+ /*
+ * To maintain atomicity of rename of the path, lease is taken on the path.
+ */
+ if (path.equals(src)) {
+ abfsLease = srcAbfsLease;
+ leaseId = srcLeaseId;
+ } else {
+ abfsLease = takeLease(path, null);
+ leaseId = abfsLease.getLeaseID();
+ }
+ } else {
+ leaseId = null;
+ }
+ boolean operated = false;
+ try {
+ copyPath(path, destinationPathForBlobPartOfRenameSrcDir, leaseId);
+ getAbfsClient().deleteBlobPath(path, leaseId, tracingContext);
+ operated = true;
+ } finally {
+ if (abfsLease != null) {
+ // If the operation is successful, cancel the timer and no need to release
+ // the lease as delete on the blob-path has taken place.
+ if (operated) {
+ abfsLease.cancelTimer();
+ } else {
+ abfsLease.free();
+ }
+ }
+ }
+ operatedBlobCount.incrementAndGet();
+ return true;
+ }
+
+ /** Copies the source path to the destination path.
+ *
+ * @param src source path
+ * @param dst destination path
+ * @param leaseId lease id for the source path
+ *
+ * @throws AzureBlobFileSystemException if server call fails
+ */
+ private void copyPath(final Path src, final Path dst, final String leaseId)
+ throws AzureBlobFileSystemException {
+ String copyId;
+ try {
+ AbfsRestOperation copyPathOp = getAbfsClient().copyBlob(src, dst, leaseId,
+ tracingContext);
+ final String progress = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_STATUS);
+ if (COPY_STATUS_SUCCESS.equalsIgnoreCase(progress)) {
+ return;
+ }
+ copyId = copyPathOp.getResult()
+ .getResponseHeader(X_MS_COPY_ID);
+ } catch (AbfsRestOperationException ex) {
+ if (ex.getStatusCode() == HttpURLConnection.HTTP_CONFLICT) {
+ AbfsRestOperation dstPathStatus = getAbfsClient().getPathStatus(
+ dst.toUri().getPath(),
+ tracingContext, null, false);
+ final String srcCopyPath = ROOT_PATH + getAbfsClient().getFileSystem()
+ + src.toUri().getPath();
+ if (dstPathStatus.getResult() != null && (srcCopyPath.equals(
+ getDstSource(dstPathStatus)))) {
+ return;
+ }
+ }
+ throw ex;
+ }
+ final long pollWait = getAbfsClient().getAbfsConfiguration()
+ .getBlobCopyProgressPollWaitMillis();
+ while (handleCopyInProgress(dst, tracingContext, copyId)
+ == BlobCopyProgress.PENDING) {
+ try {
+ Thread.sleep(pollWait);
+ } catch (InterruptedException ignored) {
+
+ }
+ }
+ }
+
+ /** Gets the source path of the copy operation.
+ *
+ * @param dstPathStatus server response for the GetBlobProperties API on the
+ * destination path.
+ *
+ * @return source path of the copy operation
+ */
+ private String getDstSource(final AbfsRestOperation dstPathStatus) {
+ try {
+ String responseHeader = dstPathStatus.getResult()
+ .getResponseHeader(X_MS_COPY_SOURCE);
+ if (responseHeader == null) {
+ return null;
+ }
+ return new URL(responseHeader).toURI().getPath();
+ } catch (URISyntaxException | MalformedURLException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Verifies if the blob copy is success or a failure or still in progress.
+ *
+ * @param dstPath path of the destination for the copying
+ * @param tracingContext object of tracingContext used for the tracing of the
+ * server calls.
+ * @param copyId id returned by server on the copy server-call. This id gets
+ * attached to blob and is returned by GetBlobProperties API on the destination.
+ *
+ * @return BlobCopyProgress indicating the status of the copy operation
+ *
+ * @throws AzureBlobFileSystemException exception returned in making server call
+ * for GetBlobProperties on the path. It can be thrown if the copyStatus is failure
+ * or is aborted.
+ */
+ @VisibleForTesting
+ public BlobCopyProgress handleCopyInProgress(final Path dstPath,
+ final TracingContext tracingContext,
+ final String copyId) throws AzureBlobFileSystemException {
+ AbfsRestOperation op = getAbfsClient().getPathStatus(dstPath.toUri().getPath(),
+ tracingContext, null, false);
+
+ if (op.getResult() != null && copyId.equals(
+ op.getResult().getResponseHeader(X_MS_COPY_ID))) {
+ final String copyStatus = op.getResult()
Review Comment:
I think this is not needed as we are already checking on op.getResult() != null and header is map, so it will handle null condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1916424425
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
Review Comment:
remove unused imports
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917838759
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/PathInformation.java:
##########
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
Review Comment:
Java doc added.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsLease.java:
##########
@@ -201,4 +238,25 @@ public int getAcquireRetryCount() {
public TracingContext getTracingContext() {
return tracingContext;
}
+
Review Comment:
Java doc added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917777307
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
+//import java.util.Collections;
+//import java.util.List;
+import java.util.Random;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsDriverException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.contracts.services.BlobAppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static java.net.HttpURLConnection.HTTP_PRECON_FAILED;
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.BLOCK_ID_LENGTH;
+//import static org.apache.hadoop.fs.azurebfs.services.AzureIngressHandler.generateBlockListXml;
+
+/**
+ * For a directory enabled for atomic-rename, before rename starts, a file with
+ * -RenamePending.json suffix is created. In this file, the states required for the
+ * rename operation are given. This file is created by {@link #preRename()} method.
+ * This is important in case the JVM process crashes during rename, the atomicity
+ * will be maintained, when the job calls {@link AzureBlobFileSystem#listStatus(Path)}
+ * or {@link AzureBlobFileSystem#getFileStatus(Path)}. On these API calls to filesystem,
+ * it will be checked if there is any RenamePending JSON file. If yes, the crashed rename
+ * operation would be resumed as per the file.
+ */
+public class RenameAtomicity {
+
+ private final TracingContext tracingContext;
+
+ private Path src, dst;
+
+ private String srcEtag;
+
+ private final AbfsBlobClient abfsClient;
+
+ private final Path renameJsonPath;
+
+ public static final String SUFFIX = "-RenamePending.json";
+
+ private int preRenameRetryCount = 0;
+
+ private int renamePendingJsonLen;
+
+ private final AbfsLease sourcePathLease;
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private static final Random RANDOM = new Random();
+
+ /**
+ * Performs pre-rename operations. Creates a file with -RenamePending.json
+ * suffix in the source parent directory. This file contains the states
+ * required for the rename operation.
+ *
+ * @param src Source path
+ * @param dst Destination path
+ * @param renameJsonPath Path of the JSON file to be created
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ */
+ public RenameAtomicity(final Path src, final Path dst,
+ final Path renameJsonPath,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient) {
+ this.src = src;
+ this.dst = dst;
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.sourcePathLease = null;
+ }
+
+ /**
+ * Resumes the rename operation from the JSON file.
+ *
+ * @param renameJsonPath Path of the JSON file
+ * @param renamePendingJsonFileLen Length of the JSON file
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ * @param sourceLease Lease of the source directory
+ */
+ public RenameAtomicity(final Path renameJsonPath,
+ final int renamePendingJsonFileLen,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient,
+ final AbfsLease sourceLease) {
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.renamePendingJsonLen = renamePendingJsonFileLen;
+ this.sourcePathLease = sourceLease;
+ }
+
+ /**
+ * Redo the rename operation from the JSON file.
+ *
+ * @throws AzureBlobFileSystemException If the redo operation fails.
+ */
+ public void redo() throws AzureBlobFileSystemException {
+ byte[] buffer = readRenamePendingJson(renameJsonPath, renamePendingJsonLen);
+ String contents = new String(buffer, Charset.defaultCharset());
+ try {
+ final RenamePendingJsonFormat renamePendingJsonFormatObj;
+ try {
+ renamePendingJsonFormatObj = objectMapper.readValue(contents,
+ RenamePendingJsonFormat.class);
+ } catch (JsonProcessingException e) {
+ return;
+ }
+ if (renamePendingJsonFormatObj != null && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getOldFolderName())
+ && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getNewFolderName())
+ && StringUtils.isNotEmpty(renamePendingJsonFormatObj.getETag())) {
+ this.src = new Path(renamePendingJsonFormatObj.getOldFolderName());
+ this.dst = new Path(renamePendingJsonFormatObj.getNewFolderName());
+ this.srcEtag = renamePendingJsonFormatObj.getETag();
+
+ BlobRenameHandler blobRenameHandler = new BlobRenameHandler(
+ this.src.toUri().getPath(), dst.toUri().getPath(),
+ abfsClient, srcEtag, true, true,
+ sourcePathLease, tracingContext);
+
+ blobRenameHandler.execute();
+ }
+ } finally {
+ deleteRenamePendingJson();
+ }
+ }
+
+ /** Read the JSON file.
+ *
+ * @param path Path of the JSON file
+ * @param len Length of the JSON file
+ * @return Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the read operation fails.
+ */
+ @VisibleForTesting
+ byte[] readRenamePendingJson(Path path, int len)
+ throws AzureBlobFileSystemException {
+ byte[] bytes = new byte[len];
+ abfsClient.read(path.toUri().getPath(), 0, bytes, 0,
+ len, null, null, null,
+ tracingContext);
+ return bytes;
+ }
+
+ /** Generate a random block ID.
+ *
+ * @return Random block ID
+ */
+ public static String generateBlockId() {
+ // PutBlock on the path.
+ byte[] blockIdByteArray = new byte[BLOCK_ID_LENGTH];
+ RANDOM.nextBytes(blockIdByteArray);
+ return new String(Base64.encodeBase64(blockIdByteArray),
+ StandardCharsets.UTF_8);
+ }
+
+ /** Create the JSON file with the contents.
+ *
+ * @param path Path of the JSON file
+ * @param bytes Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the create operation fails.
+ */
+ @VisibleForTesting
+ void createRenamePendingJson(Path path, byte[] bytes)
+ throws AzureBlobFileSystemException {
+ // PutBlob on the path.
+ AbfsRestOperation putBlobOp = abfsClient.createPath(path.toUri().getPath(),
+ true,
+ true,
+ null,
+ false,
+ null,
+ null,
+ tracingContext);
+ String eTag = extractEtagHeader(putBlobOp.getResult());
+
+ String blockId = generateBlockId();
+ AppendRequestParameters appendRequestParameters
+ = new AppendRequestParameters(0, 0,
+ bytes.length, AppendRequestParameters.Mode.APPEND_MODE, false, null,
+ abfsClient.getAbfsConfiguration().isExpectHeaderEnabled(),
+ new BlobAppendRequestParameters(blockId, eTag));
+
+ abfsClient.append(path.toUri().getPath(), bytes,
+ appendRequestParameters, null, null, tracingContext);
+
+// List<String> blockIdList = new ArrayList<>(Collections.singleton(blockId));
Review Comment:
This will go once Ingress handler changes are taken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1917776552
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
+//import java.util.Collections;
+//import java.util.List;
+import java.util.Random;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsDriverException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.contracts.services.BlobAppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static java.net.HttpURLConnection.HTTP_PRECON_FAILED;
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.BLOCK_ID_LENGTH;
+//import static org.apache.hadoop.fs.azurebfs.services.AzureIngressHandler.generateBlockListXml;
+
+/**
+ * For a directory enabled for atomic-rename, before rename starts, a file with
+ * -RenamePending.json suffix is created. In this file, the states required for the
+ * rename operation are given. This file is created by {@link #preRename()} method.
+ * This is important in case the JVM process crashes during rename, the atomicity
+ * will be maintained, when the job calls {@link AzureBlobFileSystem#listStatus(Path)}
+ * or {@link AzureBlobFileSystem#getFileStatus(Path)}. On these API calls to filesystem,
+ * it will be checked if there is any RenamePending JSON file. If yes, the crashed rename
+ * operation would be resumed as per the file.
+ */
+public class RenameAtomicity {
+
+ private final TracingContext tracingContext;
+
+ private Path src, dst;
+
+ private String srcEtag;
+
+ private final AbfsBlobClient abfsClient;
+
+ private final Path renameJsonPath;
+
+ public static final String SUFFIX = "-RenamePending.json";
+
+ private int preRenameRetryCount = 0;
+
+ private int renamePendingJsonLen;
+
+ private final AbfsLease sourcePathLease;
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private static final Random RANDOM = new Random();
+
+ /**
+ * Performs pre-rename operations. Creates a file with -RenamePending.json
+ * suffix in the source parent directory. This file contains the states
+ * required for the rename operation.
+ *
+ * @param src Source path
+ * @param dst Destination path
+ * @param renameJsonPath Path of the JSON file to be created
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ */
+ public RenameAtomicity(final Path src, final Path dst,
+ final Path renameJsonPath,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient) {
+ this.src = src;
+ this.dst = dst;
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.sourcePathLease = null;
+ }
+
+ /**
+ * Resumes the rename operation from the JSON file.
+ *
+ * @param renameJsonPath Path of the JSON file
+ * @param renamePendingJsonFileLen Length of the JSON file
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ * @param sourceLease Lease of the source directory
+ */
+ public RenameAtomicity(final Path renameJsonPath,
+ final int renamePendingJsonFileLen,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient,
+ final AbfsLease sourceLease) {
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.renamePendingJsonLen = renamePendingJsonFileLen;
+ this.sourcePathLease = sourceLease;
+ }
+
+ /**
+ * Redo the rename operation from the JSON file.
+ *
+ * @throws AzureBlobFileSystemException If the redo operation fails.
+ */
+ public void redo() throws AzureBlobFileSystemException {
+ byte[] buffer = readRenamePendingJson(renameJsonPath, renamePendingJsonLen);
+ String contents = new String(buffer, Charset.defaultCharset());
+ try {
+ final RenamePendingJsonFormat renamePendingJsonFormatObj;
+ try {
+ renamePendingJsonFormatObj = objectMapper.readValue(contents,
+ RenamePendingJsonFormat.class);
+ } catch (JsonProcessingException e) {
+ return;
+ }
+ if (renamePendingJsonFormatObj != null && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getOldFolderName())
+ && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getNewFolderName())
+ && StringUtils.isNotEmpty(renamePendingJsonFormatObj.getETag())) {
+ this.src = new Path(renamePendingJsonFormatObj.getOldFolderName());
+ this.dst = new Path(renamePendingJsonFormatObj.getNewFolderName());
+ this.srcEtag = renamePendingJsonFormatObj.getETag();
+
+ BlobRenameHandler blobRenameHandler = new BlobRenameHandler(
+ this.src.toUri().getPath(), dst.toUri().getPath(),
+ abfsClient, srcEtag, true, true,
+ sourcePathLease, tracingContext);
+
+ blobRenameHandler.execute();
+ }
+ } finally {
+ deleteRenamePendingJson();
+ }
+ }
+
+ /** Read the JSON file.
+ *
+ * @param path Path of the JSON file
+ * @param len Length of the JSON file
+ * @return Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the read operation fails.
+ */
+ @VisibleForTesting
+ byte[] readRenamePendingJson(Path path, int len)
+ throws AzureBlobFileSystemException {
+ byte[] bytes = new byte[len];
+ abfsClient.read(path.toUri().getPath(), 0, bytes, 0,
+ len, null, null, null,
+ tracingContext);
+ return bytes;
+ }
+
+ /** Generate a random block ID.
+ *
+ * @return Random block ID
+ */
+ public static String generateBlockId() {
+ // PutBlock on the path.
+ byte[] blockIdByteArray = new byte[BLOCK_ID_LENGTH];
+ RANDOM.nextBytes(blockIdByteArray);
+ return new String(Base64.encodeBase64(blockIdByteArray),
+ StandardCharsets.UTF_8);
+ }
+
+ /** Create the JSON file with the contents.
+ *
+ * @param path Path of the JSON file
+ * @param bytes Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the create operation fails.
+ */
+ @VisibleForTesting
+ void createRenamePendingJson(Path path, byte[] bytes)
+ throws AzureBlobFileSystemException {
+ // PutBlob on the path.
+ AbfsRestOperation putBlobOp = abfsClient.createPath(path.toUri().getPath(),
+ true,
+ true,
+ null,
+ false,
+ null,
+ null,
+ tracingContext);
+ String eTag = extractEtagHeader(putBlobOp.getResult());
+
+ String blockId = generateBlockId();
+ AppendRequestParameters appendRequestParameters
+ = new AppendRequestParameters(0, 0,
+ bytes.length, AppendRequestParameters.Mode.APPEND_MODE, false, null,
+ abfsClient.getAbfsConfiguration().isExpectHeaderEnabled(),
+ new BlobAppendRequestParameters(blockId, eTag));
+
+ abfsClient.append(path.toUri().getPath(), bytes,
+ appendRequestParameters, null, null, tracingContext);
+
+// List<String> blockIdList = new ArrayList<>(Collections.singleton(blockId));
+// String blockList = generateBlockListXml(blockIdList);
+ // PutBlockList on the path.
+ String blockList = "";
Review Comment:
Added this line just to pass the build. The above commented line is calling generateBlockListXml which requires ingress handler changes. Will pick the changes of ingress handler once it is merged and this line is no longer needed after that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936768361
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -782,10 +790,10 @@ public synchronized void close() throws IOException {
}
if (abfsStore.getClient().isMetricCollectionEnabled()) {
TracingContext tracingMetricContext = new TracingContext(
- clientCorrelationId,
- fileSystemId, FSOperationType.GET_ATTR, true,
- tracingHeaderFormat,
- listener, abfsCounters.toString());
+ clientCorrelationId,
Review Comment:
Taken!
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -220,7 +219,7 @@ public void initialize(URI uri, Configuration configuration)
this.setWorkingDirectory(this.getHomeDirectory());
TracingContext initFSTracingContext = new TracingContext(clientCorrelationId,
- fileSystemId, FSOperationType.INIT, tracingHeaderFormat, listener);
+ fileSystemId, FSOperationType.INIT, tracingHeaderFormat, listener);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936667844
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsPaginatedDelete.java:
##########
@@ -142,6 +142,7 @@ private void setTestUserConf(Configuration conf, String key, String value) {
*/
@Test
public void testRecursiveDeleteWithPagination() throws Exception {
+ assertTrue(getFileSystem().getAbfsStore().getClient() instanceof AbfsDfsClient);
Review Comment:
We should use `assumeBlobServiceType()` of base class. here and everywhere for cleaner code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2626967053
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 13 new or modified test files. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 39m 17s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/23/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 23 unchanged - 3 fixed = 23 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 28s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/23/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 25s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/23/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-azure in the patch failed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga. |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 34s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 128m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/23/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 90dc583c747e 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dc7932b2bafa601c9c65cbd426dea5ad9c53ccbe |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/23/testReport/ |
| Max. process+thread count | 525 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/23/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1937076854
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java:
##########
@@ -409,6 +412,24 @@ public AbfsRestOperation createPath(final String path,
return op;
}
+ public void createNonRecursivePreCheck(Path parentPath,
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1933638219
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRenameUnicode.java:
##########
@@ -26,13 +26,19 @@
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathIOException;
Review Comment:
Done
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -18,36 +18,83 @@
package org.apache.hadoop.fs.azurebfs;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.AccessDeniedException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
-import org.junit.Assert;
+import org.apache.hadoop.fs.FileSystem;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936796353
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
+//import java.util.Collections;
+//import java.util.List;
+import java.util.Random;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.commons.codec.binary.Base64;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsDriverException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException;
+import org.apache.hadoop.fs.azurebfs.contracts.services.AppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.contracts.services.BlobAppendRequestParameters;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+
+import static java.net.HttpURLConnection.HTTP_NOT_FOUND;
+import static java.net.HttpURLConnection.HTTP_PRECON_FAILED;
+import static org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.extractEtagHeader;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.BLOCK_ID_LENGTH;
+//import static org.apache.hadoop.fs.azurebfs.services.AzureIngressHandler.generateBlockListXml;
+
+/**
+ * For a directory enabled for atomic-rename, before rename starts, a file with
+ * -RenamePending.json suffix is created. In this file, the states required for the
+ * rename operation are given. This file is created by {@link #preRename()} method.
+ * This is important in case the JVM process crashes during rename, the atomicity
+ * will be maintained, when the job calls {@link AzureBlobFileSystem#listStatus(Path)}
+ * or {@link AzureBlobFileSystem#getFileStatus(Path)}. On these API calls to filesystem,
+ * it will be checked if there is any RenamePending JSON file. If yes, the crashed rename
+ * operation would be resumed as per the file.
+ */
+public class RenameAtomicity {
+
+ private final TracingContext tracingContext;
+
+ private Path src, dst;
+
+ private String srcEtag;
+
+ private final AbfsBlobClient abfsClient;
+
+ private final Path renameJsonPath;
+
+ public static final String SUFFIX = "-RenamePending.json";
+
+ private int preRenameRetryCount = 0;
+
+ private int renamePendingJsonLen;
+
+ private final AbfsLease sourcePathLease;
+
+ private final ObjectMapper objectMapper = new ObjectMapper();
+
+ private static final Random RANDOM = new Random();
+
+ /**
+ * Performs pre-rename operations. Creates a file with -RenamePending.json
+ * suffix in the source parent directory. This file contains the states
+ * required for the rename operation.
+ *
+ * @param src Source path
+ * @param dst Destination path
+ * @param renameJsonPath Path of the JSON file to be created
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ */
+ public RenameAtomicity(final Path src, final Path dst,
+ final Path renameJsonPath,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient) {
+ this.src = src;
+ this.dst = dst;
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.sourcePathLease = null;
+ }
+
+ /**
+ * Resumes the rename operation from the JSON file.
+ *
+ * @param renameJsonPath Path of the JSON file
+ * @param renamePendingJsonFileLen Length of the JSON file
+ * @param tracingContext Tracing context
+ * @param srcEtag ETag of the source directory
+ * @param abfsClient AbfsClient instance
+ * @param sourceLease Lease of the source directory
+ */
+ public RenameAtomicity(final Path renameJsonPath,
+ final int renamePendingJsonFileLen,
+ TracingContext tracingContext,
+ final String srcEtag,
+ final AbfsClient abfsClient,
+ final AbfsLease sourceLease) {
+ this.abfsClient = (AbfsBlobClient) abfsClient;
+ this.renameJsonPath = renameJsonPath;
+ this.tracingContext = tracingContext;
+ this.srcEtag = srcEtag;
+ this.renamePendingJsonLen = renamePendingJsonFileLen;
+ this.sourcePathLease = sourceLease;
+ }
+
+ /**
+ * Redo the rename operation from the JSON file.
+ *
+ * @throws AzureBlobFileSystemException If the redo operation fails.
+ */
+ public void redo() throws AzureBlobFileSystemException {
+ byte[] buffer = readRenamePendingJson(renameJsonPath, renamePendingJsonLen);
+ String contents = new String(buffer, Charset.defaultCharset());
+ try {
+ final RenamePendingJsonFormat renamePendingJsonFormatObj;
+ try {
+ renamePendingJsonFormatObj = objectMapper.readValue(contents,
+ RenamePendingJsonFormat.class);
+ } catch (JsonProcessingException e) {
+ return;
+ }
+ if (renamePendingJsonFormatObj != null && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getOldFolderName())
+ && StringUtils.isNotEmpty(
+ renamePendingJsonFormatObj.getNewFolderName())
+ && StringUtils.isNotEmpty(renamePendingJsonFormatObj.getETag())) {
+ this.src = new Path(renamePendingJsonFormatObj.getOldFolderName());
+ this.dst = new Path(renamePendingJsonFormatObj.getNewFolderName());
+ this.srcEtag = renamePendingJsonFormatObj.getETag();
+
+ BlobRenameHandler blobRenameHandler = new BlobRenameHandler(
+ this.src.toUri().getPath(), dst.toUri().getPath(),
+ abfsClient, srcEtag, true, true,
+ sourcePathLease, tracingContext);
+
+ blobRenameHandler.execute();
+ }
+ } finally {
+ deleteRenamePendingJson();
+ }
+ }
+
+ /** Read the JSON file.
+ *
+ * @param path Path of the JSON file
+ * @param len Length of the JSON file
+ * @return Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the read operation fails.
+ */
+ @VisibleForTesting
+ byte[] readRenamePendingJson(Path path, int len)
+ throws AzureBlobFileSystemException {
+ byte[] bytes = new byte[len];
+ abfsClient.read(path.toUri().getPath(), 0, bytes, 0,
+ len, null, null, null,
+ tracingContext);
+ return bytes;
+ }
+
+ /** Generate a random block ID.
+ *
+ * @return Random block ID
+ */
+ public static String generateBlockId() {
+ // PutBlock on the path.
+ byte[] blockIdByteArray = new byte[BLOCK_ID_LENGTH];
+ RANDOM.nextBytes(blockIdByteArray);
+ return new String(Base64.encodeBase64(blockIdByteArray),
+ StandardCharsets.UTF_8);
+ }
+
+ /** Create the JSON file with the contents.
+ *
+ * @param path Path of the JSON file
+ * @param bytes Contents of the JSON file
+ * @throws AzureBlobFileSystemException If the create operation fails.
+ */
+ @VisibleForTesting
+ void createRenamePendingJson(Path path, byte[] bytes)
+ throws AzureBlobFileSystemException {
+ // PutBlob on the path.
+ AbfsRestOperation putBlobOp = abfsClient.createPath(path.toUri().getPath(),
+ true,
+ true,
+ null,
+ false,
+ null,
+ null,
+ tracingContext);
+ String eTag = extractEtagHeader(putBlobOp.getResult());
+
+ String blockId = generateBlockId();
+ AppendRequestParameters appendRequestParameters
+ = new AppendRequestParameters(0, 0,
+ bytes.length, AppendRequestParameters.Mode.APPEND_MODE, false, null,
+ abfsClient.getAbfsConfiguration().isExpectHeaderEnabled(),
+ new BlobAppendRequestParameters(blockId, eTag));
+
+ abfsClient.append(path.toUri().getPath(), bytes,
+ appendRequestParameters, null, null, tracingContext);
+
+// List<String> blockIdList = new ArrayList<>(Collections.singleton(blockId));
+// String blockList = generateBlockListXml(blockIdList);
+ // PutBlockList on the path.
+ String blockList = "";
Review Comment:
This is removed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936793771
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelete.java:
##########
@@ -314,4 +347,278 @@ public void deleteBlobDirParallelThreadToDeleteOnDifferentTracingContext()
fs.delete(new Path("/testDir"), true);
fs.close();
}
+
+ private void assumeBlobClient() throws IOException {
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936113007
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -341,7 +360,7 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
abfsUriQueryBuilder.addQuery(QUERY_PARAM_DELIMITER, FORWARD_SLASH);
}
abfsUriQueryBuilder.addQuery(QUERY_PARAM_MAX_RESULTS, String.valueOf(listMaxResults));
- appendSASTokenToQuery(relativePath, SASTokenProvider.FIXED_SAS_STORE_OPERATION, abfsUriQueryBuilder);
+ appendSASTokenToQuery(relativePath, SASTokenProvider.LIST_OPERATION, abfsUriQueryBuilder);
Review Comment:
change to FIXED_SAS_STORE_OPERATION
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936114934
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1556,6 +1721,128 @@ public static String getDirectoryQueryParameter(final String path) {
return directory;
}
+ public boolean isAtomicRenameKey(String key) {
Review Comment:
missing javadocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1936796925
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/RenameAtomicity.java:
##########
@@ -0,0 +1,343 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+
+//import java.util.ArrayList;
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2597890317
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 18m 7s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 26s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 0s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | hadoop-tools/hadoop-azure: The patch generated 0 new + 13 unchanged - 3 fixed = 13 total (was 16) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | javadoc | 0m 26s | | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 24s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 40s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 152m 57s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 0fe1b624c68c 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e2ddbdff0b169b4e11676032c94a39a9c037f6d7 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/9/testReport/ |
| Max. process+thread count | 615 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1939027360
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -530,45 +541,31 @@ public boolean rename(final Path src, final Path dst) throws IOException {
return tryGetFileStatus(qualifiedSrcPath, tracingContext) != null;
}
- FileStatus dstFileStatus = null;
+ FileStatus dstFileStatus = tryGetFileStatus(qualifiedDstPath, tracingContext);
Review Comment:
We are changing a long existing code here, lets try to make sure any corner cases are not ignored.
All the tests should run, if you think any scenario is missed by tests please add it.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -203,7 +203,7 @@ private AbfsClient(final URL baseUrl,
if (encryptionContextProvider != null) {
this.encryptionContextProvider = encryptionContextProvider;
- xMsVersion = ApiVersion.APR_10_2021; // will be default once server change deployed
+ xMsVersion = ApiVersion.AUG_03_2023; // will be default once server change deployed
Review Comment:
Nit: Add a comment why this version upgrade is needed.
cc: @anmolanmol1234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#issuecomment-2630949601
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 21 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 53s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/32/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 23 unchanged - 3 fixed = 24 total (was 26) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 28s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/32/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 10 unchanged - 1 fixed = 11 total (was 11) |
| -1 :x: | javadoc | 0m 26s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/32/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 1 new + 10 unchanged - 1 fixed = 11 total (was 11) |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 38s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 131m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/32/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7265 |
| JIRA Issue | HADOOP-19233 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux bb6037c9075f 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8a4c1cf408ac3802a65c662a401536525167d598 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/32/testReport/ |
| Max. process+thread count | 602 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7265/32/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19233: ABFS: [FnsOverBlob] Implementing Rename and Delete APIs over Blob Endpoint [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7265:
URL: https://github.com/apache/hadoop/pull/7265#discussion_r1938967651
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -535,34 +541,20 @@ public boolean rename(final Path src, final Path dst) throws IOException {
return tryGetFileStatus(qualifiedSrcPath, tracingContext) != null;
}
- FileStatus dstFileStatus = null;
+ FileStatus dstFileStatus = tryGetFileStatus(qualifiedDstPath, tracingContext);
+ Path adjustedDst = dst;
if (qualifiedSrcPath.equals(qualifiedDstPath)) {
- // rename to itself
- // - if it doesn't exist, return false
- // - if it is file, return true
- // - if it is dir, return false.
- dstFileStatus = tryGetFileStatus(qualifiedDstPath, tracingContext);
- if (dstFileStatus == null) {
+ return dstFileStatus != null && !dstFileStatus.isDirectory();
+ } else if (dstFileStatus != null) {
Review Comment:
dont see a need for this else if
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org