You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "anujmodi2021 (via GitHub)" <gi...@apache.org> on 2025/02/21 05:01:05 UTC
[PR] Hadoop 19234 followup [hadoop]
anujmodi2021 opened a new pull request, #7421:
URL: https://github.com/apache/hadoop/pull/7421
### Description of PR
List Call Optimizations and Negative Scenario testing
### How was this patch tested?
### For code changes:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014021214
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsCustomEncryption.java:
##########
@@ -301,10 +302,10 @@ private AbfsRestOperation callOperation(AzureBlobFileSystem fs,
*/
FileStatus status = fs.listStatus(testPath)[0];
Assertions.assertThat(status)
Review Comment:
we can add assertion statement here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014013003
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
Review Comment:
Taken as per Manish's suggestions above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2004818573
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.services;
Review Comment:
Hmm.
Not very sure. All other classes related to List parsing were there only so kept it there.
But again this is used in public interfaces. What do you suggest will be a good place for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013943436
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
Review Comment:
Good Suggestion
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014014171
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java:
##########
@@ -1464,20 +1472,41 @@ public Hashtable<String, String> getXMSProperties(AbfsHttpOperation result)
/**
* Parse the list file response from DFS ListPath API in Json format
- * @param stream InputStream contains the list results.
- * @throws IOException if parsing fails.
+ * @param result InputStream contains the list results.
+ * @param uri to be used for path conversion.
+ * @return {@link ListResponseData}. containing listing response.
+ * @throws AzureBlobFileSystemException if parsing fails.
*/
@Override
- public ListResultSchema parseListPathResults(final InputStream stream) throws IOException {
- DfsListResultSchema listResultSchema;
- try {
- final ObjectMapper objectMapper = new ObjectMapper();
- listResultSchema = objectMapper.readValue(stream, DfsListResultSchema.class);
+ public ListResponseData parseListPathResults(AbfsHttpOperation result, URI uri) throws AzureBlobFileSystemException {
+ try (InputStream listResultInputStream = result.getListResultStream()) {
+ DfsListResultSchema listResultSchema;
+ try {
+ final ObjectMapper objectMapper = new ObjectMapper();
+ listResultSchema = objectMapper.readValue(listResultInputStream,
+ DfsListResultSchema.class);
+ result.setListResultSchema(listResultSchema);
+ LOG.debug("ListPath listed {} paths with {} as continuation token",
+ listResultSchema.paths().size(),
+ getContinuationFromResponse(result));
+ } catch (IOException ex) {
+ throw new AbfsDriverException(ex);
+ }
+
+ List<FileStatus> fileStatuses = new ArrayList<>();
+ for (DfsListResultEntrySchema entry : listResultSchema.paths()) {
+ fileStatuses.add(getVersionedFileStatusFromEntry(entry, uri));
+ }
+ ListResponseData listResponseData = new ListResponseData();
+ listResponseData.setFileStatusList(fileStatuses);
+ listResponseData.setRenamePendingJsonPaths(null);
+ listResponseData.setContinuationToken(
+ getContinuationFromResponse(result));
+ return listResponseData;
} catch (IOException ex) {
LOG.error("Unable to deserialize list results", ex);
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014036287
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java:
##########
@@ -343,4 +319,174 @@ public void testRenameTrailingPeriodFile() throws IOException {
assertTrue("Attempt to create file that ended with a dot should"
+ " throw IllegalArgumentException", exceptionThrown);
}
+
+
+
+ /**
+ * Test to verify that listStatus returns the correct file status all types
+ * of paths viz. implicit, explicit, file.
+ * @throws Exception if there is an error or test assertions fails.
+ */
+ @Test
+ public void testListStatusWithImplicitExplicitChildren() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path root = new Path(ROOT_PATH);
+
+ // Create an implicit directory under root
+ Path dir = new Path("a");
+ Path fileInsideDir = new Path("a/file");
+ createAzCopyFolder(dir);
+
+ // Assert that implicit directory is returned
+ FileStatus[] fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(1);
+ assertImplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+
+ // Create a marker blob for the directory.
+ fs.create(fileInsideDir);
+
+ // Assert that only one entry of explicit directory is returned
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(1);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+
+ // Create a file under root
+ Path file1 = new Path("b");
+ fs.create(file1);
+
+ // Assert that two entries are returned in alphabetic order.
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(2);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+ assertFileFileStatus(fileStatuses[1], fs.makeQualified(file1));
+
+ // Create another implicit directory under root.
+ Path dir2 = new Path("c");
+ createAzCopyFolder(dir2);
+
+ // Assert that three entries are returned in alphabetic order.
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(3);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+ assertFileFileStatus(fileStatuses[1], fs.makeQualified(file1));
+ assertImplicitDirectoryFileStatus(fileStatuses[2], fs.makeQualified(dir2));
+ }
+
+ /**
+ * Test to verify that listStatus returns the correct file status when called on an implicit path
+ * @throws Exception if there is an error or test assertions fails.
+ */
+ @Test
+ public void testListStatusOnImplicitDirectoryPath() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path implicitPath = new Path("/implicitDir");
+ createAzCopyFolder(implicitPath);
+
+ FileStatus[] statuses = fs.listStatus(implicitPath);
+ Assertions.assertThat(statuses.length).isGreaterThanOrEqualTo(1);
+ assertImplicitDirectoryFileStatus(statuses[0], fs.makeQualified(statuses[0].getPath()));
+
+ FileStatus[] statuses1 = fs.listStatus(new Path(statuses[0].getPath().toString()));
+ Assertions.assertThat(statuses1.length).isGreaterThanOrEqualTo(1);
+ assertFileFileStatus(statuses1[0], fs.makeQualified(statuses1[0].getPath()));
+ }
+
+ @Test
+ public void testListStatusOnEmptyDirectory() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path emptyDir = new Path("/emptyDir");
+ fs.mkdirs(emptyDir);
+
+ FileStatus[] statuses = fs.listStatus(emptyDir);
+ Assertions.assertThat(statuses.length).isEqualTo(0);
+ }
+
+ @Test
+ public void testContinuationTokenAcrossListStatus() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ fs.listStatus(path);
+
+ ListResponseData listResponseData = fs.getAbfsStore().getClient().listPath(
+ "/testContinuationToken", false, 1, null, getTestTracingContext(fs, true),
+ fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData.getContinuationToken()).isNotNull();
+ Assertions.assertThat(listResponseData.getFileStatusList()).hasSize(1);
+
+ ListResponseData listResponseData1 = fs.getAbfsStore().getClient().listPath(
+ "/testContinuationToken", false, 1, listResponseData.getContinuationToken(), getTestTracingContext(fs, true),
+ fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData1.getContinuationToken()).isNull();
+ Assertions.assertThat(listResponseData1.getFileStatusList()).hasSize(1);
+ }
+
+ @Test
+ public void testInvalidContinuationToken() throws Exception {
+ assumeHnsDisabled();
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testInvalidContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ intercept(AbfsRestOperationException.class,
+ () -> fs.getAbfsStore().getClient().listPath(
+ "/testInvalidContinuationToken", false, 1, "invalidToken",
+ getTestTracingContext(fs, true), fs.getAbfsStore().getUri()));
+ }
+
+ @Test
+ public void testEmptyContinuationToken() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testInvalidContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ ListResponseData listResponseData = fs.getAbfsStore().getClient().listPath(
+ "/testInvalidContinuationToken", false, 1, "",
+ getTestTracingContext(fs, true), fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData.getContinuationToken()).isNotNull();
+ Assertions.assertThat(listResponseData.getFileStatusList()).hasSize(1);
+ }
+
+ private void assertFileFileStatus(final FileStatus fileStatus,
+ final Path qualifiedPath) {
+ Assertions.assertThat(fileStatus.getPath()).isEqualTo(qualifiedPath);
+ Assertions.assertThat(fileStatus.isFile()).isEqualTo(true);
+ Assertions.assertThat(fileStatus.isDirectory()).isEqualTo(false);
+ Assertions.assertThat(fileStatus.getModificationTime()).isNotEqualTo(0);
+ }
+
+ private void assertImplicitDirectoryFileStatus(final FileStatus fileStatus,
+ final Path qualifiedPath) throws Exception {
+ assertDirectoryFileStatus(fileStatus, qualifiedPath);
+ DirectoryStateHelper.isImplicitDirectory(qualifiedPath, getFileSystem(),
+ getTestTracingContext(getFileSystem(), true));
+ Assertions.assertThat(fileStatus.getModificationTime()).isEqualTo(0);
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014192835
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -404,13 +403,8 @@ public ListResponseData listPath(final String relativePath, final boolean recurs
BlobListResultSchema listResultSchema = getListResultSchemaFromPathStatus(relativePath, pathStatus);
LOG.debug("ListBlob attempted on a file path. Returning file status.");
List<FileStatus> fileStatusList = new ArrayList<>();
- Map<Path, Integer> renamePendingJsonPaths = new HashMap<>();
for (BlobListResultEntrySchema entry : listResultSchema.paths()) {
- if (isRenamePendingJsonPathEntry(entry)) {
- renamePendingJsonPaths.put(entry.path(), entry.contentLength().intValue());
- } else {
- fileStatusList.add(getVersionedFileStatusFromEntry(entry, uri));
- }
+ fileStatusList.add(getVersionedFileStatusFromEntry(entry, uri));
Review Comment:
This is the code path where handling for listStatus on a file path is done. I checked on wasb if the listing is done on a filepath atomicity related checks are not performed and rename redo is not triggered. Keeping same behavior here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2695179276
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 26m 34s | | trunk passed |
| +1 :green_heart: | compile | 0m 23s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 19s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 1s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 12s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 11s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/4/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 17 new + 12 unchanged - 0 fixed = 29 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| -1 :x: | javadoc | 0m 14s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/4/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10) |
| -1 :x: | javadoc | 0m 15s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/4/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10) |
| +1 :green_heart: | spotbugs | 0m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 33s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 23s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not generate ASF License warnings. |
| | | 84m 22s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5e009114837d 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ec1419b47587389823a2b6175322bb8541a25e34 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/4/testReport/ |
| Max. process+thread count | 559 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r1982792319
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1903,39 +1916,57 @@ private List<AbfsHttpHeader> getMetadataHeadersList(final Hashtable<String, Stri
* This is to handle duplicate listing entries returned by Blob Endpoint for
* implicit paths that also has a marker file created for them.
* This will retain entry corresponding to marker file and remove the BlobPrefix entry.
+ * This will also filter out all the rename pending json files in listing output.
* @param listResultSchema List of entries returned by Blob Endpoint.
+ * @param uri URI to be used for path conversion.
* @return List of entries after removing duplicates.
*/
- private BlobListResultSchema removeDuplicateEntries(BlobListResultSchema listResultSchema) {
- List<BlobListResultEntrySchema> uniqueEntries = new ArrayList<>();
+ private ListResponseData filterDuplicateEntriesAndRenamePendingFiles(
+ BlobListResultSchema listResultSchema, URI uri) throws IOException {
+ List<FileStatus> fileStatuses = new ArrayList<>();
+ Map<Path, Integer> renamePendingJsonPaths = new HashMap<>();
TreeMap<String, BlobListResultEntrySchema> nameToEntryMap = new TreeMap<>();
for (BlobListResultEntrySchema entry : listResultSchema.paths()) {
if (StringUtils.isNotEmpty(entry.eTag())) {
// This is a blob entry. It is either a file or a marker blob.
// In both cases we will add this.
nameToEntryMap.put(entry.name(), entry);
+ fileStatuses.add(getVersionedFileStatusFromEntry(entry, uri));
+
+ if (isRenamePendingJsonPathEntry(entry)) {
+ renamePendingJsonPaths.put(entry.path(), entry.contentLength().intValue());
+ }
} else {
// This is a BlobPrefix entry. It is a directory with file inside
// This might have already been added as a marker blob.
if (!nameToEntryMap.containsKey(entry.name())) {
nameToEntryMap.put(entry.name(), entry);
+ fileStatuses.add(getVersionedFileStatusFromEntry(entry, uri));
}
}
}
- uniqueEntries.addAll(nameToEntryMap.values());
- listResultSchema.withPaths(uniqueEntries);
- return listResultSchema;
+ ListResponseData listResponseData = new ListResponseData();
+ listResponseData.setFileStatusList(fileStatuses);
+ listResponseData.setRenamePendingJsonPaths(renamePendingJsonPaths);
+ listResponseData.setContinuationToken(listResultSchema.getNextMarker());
+ return listResponseData;
+ }
+
+ private boolean isRenamePendingJsonPathEntry(BlobListResultEntrySchema entry) {
Review Comment:
Can be simplified to this :- private boolean isRenamePendingJsonPathEntry(BlobListResultEntrySchema entry) {
String path = entry.path() != null ? entry.path().toUri().getPath() : null;
return path != null && !entry.path().isRoot() && isAtomicRenameKey(path) && path.endsWith(RenameAtomicity.SUFFIX);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013939076
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
+ try {
+ primaryUserGroup = UserGroupInformation.getCurrentUser().getPrimaryGroupName();
+ } catch (IOException ex) {
+ LOG.error("Failed to get primary group for {}, using user name as primary group name",
+ getPrimaryUser());
+ primaryUserGroup = getPrimaryUser();
+ }
+ } else {
+ //Provide a default group name
+ primaryUserGroup = getPrimaryUser();
+ }
+ return primaryUserGroup;
+ }
+
+ private String getPrimaryUser() throws AzureBlobFileSystemException {
+ try {
+ return UserGroupInformation.getCurrentUser().getUserName();
+ } catch (IOException ex) {
+ throw new AbfsDriverException(ex);
+ }
+ }
+
+ /**
+ * Creates a VersionedFileStatus object from the ListResultEntrySchema.
+ * @param entry ListResultEntrySchema object.
+ * @param uri to be used for the path conversion.
+ * @return VersionedFileStatus object.
+ * @throws AzureBlobFileSystemException if transformation fails.
+ */
+ protected VersionedFileStatus getVersionedFileStatusFromEntry(
+ ListResultEntrySchema entry, URI uri) throws AzureBlobFileSystemException {
+ long blockSize = abfsConfiguration.getAzureBlockSize();
+ final String owner, group;
+ try{
+ if (identityTransformer != null) {
+ owner = identityTransformer.transformIdentityForGetRequest(
+ entry.owner(), true, getPrimaryUser());
+ group = identityTransformer.transformIdentityForGetRequest(
+ entry.group(), false, getPrimaryUserGroup());
+ } else {
+ owner = null;
Review Comment:
Makes sense.
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014016042
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java:
##########
@@ -343,4 +319,174 @@ public void testRenameTrailingPeriodFile() throws IOException {
assertTrue("Attempt to create file that ended with a dot should"
+ " throw IllegalArgumentException", exceptionThrown);
}
+
+
+
+ /**
+ * Test to verify that listStatus returns the correct file status all types
+ * of paths viz. implicit, explicit, file.
+ * @throws Exception if there is an error or test assertions fails.
+ */
+ @Test
+ public void testListStatusWithImplicitExplicitChildren() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path root = new Path(ROOT_PATH);
+
+ // Create an implicit directory under root
+ Path dir = new Path("a");
+ Path fileInsideDir = new Path("a/file");
+ createAzCopyFolder(dir);
+
+ // Assert that implicit directory is returned
+ FileStatus[] fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(1);
+ assertImplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+
+ // Create a marker blob for the directory.
+ fs.create(fileInsideDir);
+
+ // Assert that only one entry of explicit directory is returned
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(1);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+
+ // Create a file under root
+ Path file1 = new Path("b");
+ fs.create(file1);
+
+ // Assert that two entries are returned in alphabetic order.
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(2);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+ assertFileFileStatus(fileStatuses[1], fs.makeQualified(file1));
+
+ // Create another implicit directory under root.
+ Path dir2 = new Path("c");
+ createAzCopyFolder(dir2);
+
+ // Assert that three entries are returned in alphabetic order.
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(3);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+ assertFileFileStatus(fileStatuses[1], fs.makeQualified(file1));
+ assertImplicitDirectoryFileStatus(fileStatuses[2], fs.makeQualified(dir2));
+ }
+
+ /**
+ * Test to verify that listStatus returns the correct file status when called on an implicit path
+ * @throws Exception if there is an error or test assertions fails.
+ */
+ @Test
+ public void testListStatusOnImplicitDirectoryPath() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path implicitPath = new Path("/implicitDir");
+ createAzCopyFolder(implicitPath);
+
+ FileStatus[] statuses = fs.listStatus(implicitPath);
+ Assertions.assertThat(statuses.length).isGreaterThanOrEqualTo(1);
+ assertImplicitDirectoryFileStatus(statuses[0], fs.makeQualified(statuses[0].getPath()));
+
+ FileStatus[] statuses1 = fs.listStatus(new Path(statuses[0].getPath().toString()));
+ Assertions.assertThat(statuses1.length).isGreaterThanOrEqualTo(1);
+ assertFileFileStatus(statuses1[0], fs.makeQualified(statuses1[0].getPath()));
+ }
+
+ @Test
+ public void testListStatusOnEmptyDirectory() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path emptyDir = new Path("/emptyDir");
+ fs.mkdirs(emptyDir);
+
+ FileStatus[] statuses = fs.listStatus(emptyDir);
+ Assertions.assertThat(statuses.length).isEqualTo(0);
+ }
+
+ @Test
+ public void testContinuationTokenAcrossListStatus() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ fs.listStatus(path);
+
+ ListResponseData listResponseData = fs.getAbfsStore().getClient().listPath(
+ "/testContinuationToken", false, 1, null, getTestTracingContext(fs, true),
+ fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData.getContinuationToken()).isNotNull();
+ Assertions.assertThat(listResponseData.getFileStatusList()).hasSize(1);
+
+ ListResponseData listResponseData1 = fs.getAbfsStore().getClient().listPath(
+ "/testContinuationToken", false, 1, listResponseData.getContinuationToken(), getTestTracingContext(fs, true),
+ fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData1.getContinuationToken()).isNull();
+ Assertions.assertThat(listResponseData1.getFileStatusList()).hasSize(1);
+ }
+
+ @Test
+ public void testInvalidContinuationToken() throws Exception {
+ assumeHnsDisabled();
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testInvalidContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ intercept(AbfsRestOperationException.class,
+ () -> fs.getAbfsStore().getClient().listPath(
+ "/testInvalidContinuationToken", false, 1, "invalidToken",
+ getTestTracingContext(fs, true), fs.getAbfsStore().getUri()));
+ }
+
+ @Test
+ public void testEmptyContinuationToken() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testInvalidContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ ListResponseData listResponseData = fs.getAbfsStore().getClient().listPath(
+ "/testInvalidContinuationToken", false, 1, "",
+ getTestTracingContext(fs, true), fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData.getContinuationToken()).isNotNull();
+ Assertions.assertThat(listResponseData.getFileStatusList()).hasSize(1);
+ }
+
+ private void assertFileFileStatus(final FileStatus fileStatus,
Review Comment:
FileStatus is for object and I want to convey here that this File Status denoted a File
But I can see how it can be confusing though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014014817
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java:
##########
@@ -383,7 +388,8 @@ final void parseResponse(final byte[] buffer,
// consume the input stream to release resources
int totalBytesRead = 0;
- try (InputStream stream = getContentInputStream()) {
+ try {
Review Comment:
Its still there but inside respective clients
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014006565
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.services;
Review Comment:
Moved to service
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2754672693
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 46s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 8s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 54s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 0s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 132m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux f5d50c8ccc68 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 972889915dbb4f9c53162838f56f6bbaeda2431e |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/12/testReport/ |
| Max. process+thread count | 529 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013907557
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1277,57 +1275,14 @@ public String listStatus(final Path path, final String startFrom,
do {
try (AbfsPerfInfo perfInfo = startTracking("listStatus", "listPath")) {
- AbfsRestOperation op = listingClient.listPath(relativePath, false,
- abfsConfiguration.getListMaxResults(), continuation,
- tracingContext);
+ ListResponseData listResponseData = listingClient.listPath(relativePath,
Review Comment:
ListResponseData itself will never be null.
Its members can be null. Will add null check for its member where ever applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2739497784
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 47s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 19s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 40s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/9/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 1s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 137m 48s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux a54f24fbde59 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5b1374cc37c7dec7f0703a3229b25e6b07c351c7 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/9/testReport/ |
| Max. process+thread count | 523 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2730515526
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 18m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 56s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 24s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 46s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 1s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 154m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux ace099d34769 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 10163c81b7a2b4eecd1458b2e081e3681fc00872 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/6/testReport/ |
| Max. process+thread count | 528 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2004848126
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1903,39 +1916,57 @@ private List<AbfsHttpHeader> getMetadataHeadersList(final Hashtable<String, Stri
* This is to handle duplicate listing entries returned by Blob Endpoint for
* implicit paths that also has a marker file created for them.
* This will retain entry corresponding to marker file and remove the BlobPrefix entry.
+ * This will also filter out all the rename pending json files in listing output.
* @param listResultSchema List of entries returned by Blob Endpoint.
+ * @param uri URI to be used for path conversion.
* @return List of entries after removing duplicates.
*/
- private BlobListResultSchema removeDuplicateEntries(BlobListResultSchema listResultSchema) {
- List<BlobListResultEntrySchema> uniqueEntries = new ArrayList<>();
+ private ListResponseData filterDuplicateEntriesAndRenamePendingFiles(
+ BlobListResultSchema listResultSchema, URI uri) throws IOException {
+ List<FileStatus> fileStatuses = new ArrayList<>();
+ Map<Path, Integer> renamePendingJsonPaths = new HashMap<>();
TreeMap<String, BlobListResultEntrySchema> nameToEntryMap = new TreeMap<>();
for (BlobListResultEntrySchema entry : listResultSchema.paths()) {
if (StringUtils.isNotEmpty(entry.eTag())) {
// This is a blob entry. It is either a file or a marker blob.
// In both cases we will add this.
nameToEntryMap.put(entry.name(), entry);
+ fileStatuses.add(getVersionedFileStatusFromEntry(entry, uri));
+
+ if (isRenamePendingJsonPathEntry(entry)) {
+ renamePendingJsonPaths.put(entry.path(), entry.contentLength().intValue());
+ }
} else {
// This is a BlobPrefix entry. It is a directory with file inside
// This might have already been added as a marker blob.
if (!nameToEntryMap.containsKey(entry.name())) {
nameToEntryMap.put(entry.name(), entry);
+ fileStatuses.add(getVersionedFileStatusFromEntry(entry, uri));
}
}
}
- uniqueEntries.addAll(nameToEntryMap.values());
- listResultSchema.withPaths(uniqueEntries);
- return listResultSchema;
+ ListResponseData listResponseData = new ListResponseData();
+ listResponseData.setFileStatusList(fileStatuses);
+ listResponseData.setRenamePendingJsonPaths(renamePendingJsonPaths);
+ listResponseData.setContinuationToken(listResultSchema.getNextMarker());
+ return listResponseData;
+ }
+
+ private boolean isRenamePendingJsonPathEntry(BlobListResultEntrySchema entry) {
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r1982759645
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.services;
Review Comment:
Is this class expected to be in the contracts folder ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013945474
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
+ try {
+ primaryUserGroup = UserGroupInformation.getCurrentUser().getPrimaryGroupName();
+ } catch (IOException ex) {
+ LOG.error("Failed to get primary group for {}, using user name as primary group name",
+ getPrimaryUser());
+ primaryUserGroup = getPrimaryUser();
+ }
+ } else {
+ //Provide a default group name
+ primaryUserGroup = getPrimaryUser();
+ }
+ return primaryUserGroup;
+ }
+
+ private String getPrimaryUser() throws AzureBlobFileSystemException {
Review Comment:
Added
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013947755
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -285,6 +298,18 @@ private AbfsClient(final URL baseUrl,
metricIdlePeriod);
}
this.abfsMetricUrl = abfsConfiguration.getMetricUri();
+
+ final Class<? extends IdentityTransformerInterface> identityTransformerClass =
+ abfsConfiguration.getRawConfiguration().getClass(FS_AZURE_IDENTITY_TRANSFORM_CLASS, IdentityTransformer.class,
+ IdentityTransformerInterface.class);
+ try {
+ this.identityTransformer =
+ identityTransformerClass.getConstructor(Configuration.class).newInstance(abfsConfiguration.getRawConfiguration());
+ } catch (IllegalAccessException | InstantiationException | IllegalArgumentException
+ | InvocationTargetException | NoSuchMethodException e) {
+ throw new IOException(e);
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014023099
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -1780,19 +1782,19 @@ private Configuration createConfig(String producerQueueSize, String consumerMaxL
private void validateRename(AzureBlobFileSystem fs, Path src, Path dst,
boolean isSrcExist, boolean isDstExist, boolean isJsonExist)
throws IOException {
- Assertions.assertThat(fs.exists(dst))
- .describedAs("Renamed Destination directory should exist.")
- .isEqualTo(isDstExist);
Assertions.assertThat(fs.exists(new Path(src.getParent(), src.getName() + SUFFIX)))
.describedAs("Renamed Pending Json file should exist.")
.isEqualTo(isJsonExist);
Assertions.assertThat(fs.exists(src))
- .describedAs("Renamed Destination directory should exist.")
+ .describedAs("Renamed Source directory should exist.")
.isEqualTo(isSrcExist);
+ Assertions.assertThat(fs.exists(dst))
+ .describedAs("Renamed Destination directory should exist.")
+ .isEqualTo(isDstExist);
}
/**
- * Test the renaming of a directory with different parallelism configurations.
+ * Test the renaming of a directory with different parallelism configurations.c
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013911075
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.services;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation;
+
+/**
+ * This class is used to hold the response data for list operations.
+ * It contains a list of FileStatus objects, a map of rename pending JSON paths,
+ * continuation token and the executed REST operation.
+ */
+public class ListResponseData {
+
+ private List<FileStatus> fileStatusList;
+ private Map<Path, Integer> renamePendingJsonPaths;
Review Comment:
I think its a good suggestion to keep it long only. Everywhere we parse content lenght as long only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013970360
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java:
##########
@@ -343,4 +319,174 @@ public void testRenameTrailingPeriodFile() throws IOException {
assertTrue("Attempt to create file that ended with a dot should"
+ " throw IllegalArgumentException", exceptionThrown);
}
+
+
+
+ /**
+ * Test to verify that listStatus returns the correct file status all types
+ * of paths viz. implicit, explicit, file.
+ * @throws Exception if there is an error or test assertions fails.
+ */
+ @Test
+ public void testListStatusWithImplicitExplicitChildren() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path root = new Path(ROOT_PATH);
+
+ // Create an implicit directory under root
+ Path dir = new Path("a");
+ Path fileInsideDir = new Path("a/file");
+ createAzCopyFolder(dir);
+
+ // Assert that implicit directory is returned
+ FileStatus[] fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(1);
+ assertImplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+
+ // Create a marker blob for the directory.
+ fs.create(fileInsideDir);
+
+ // Assert that only one entry of explicit directory is returned
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(1);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+
+ // Create a file under root
+ Path file1 = new Path("b");
+ fs.create(file1);
+
+ // Assert that two entries are returned in alphabetic order.
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(2);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+ assertFileFileStatus(fileStatuses[1], fs.makeQualified(file1));
+
+ // Create another implicit directory under root.
+ Path dir2 = new Path("c");
+ createAzCopyFolder(dir2);
+
+ // Assert that three entries are returned in alphabetic order.
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(3);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+ assertFileFileStatus(fileStatuses[1], fs.makeQualified(file1));
+ assertImplicitDirectoryFileStatus(fileStatuses[2], fs.makeQualified(dir2));
+ }
+
+ /**
+ * Test to verify that listStatus returns the correct file status when called on an implicit path
+ * @throws Exception if there is an error or test assertions fails.
+ */
+ @Test
+ public void testListStatusOnImplicitDirectoryPath() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path implicitPath = new Path("/implicitDir");
+ createAzCopyFolder(implicitPath);
+
+ FileStatus[] statuses = fs.listStatus(implicitPath);
+ Assertions.assertThat(statuses.length).isGreaterThanOrEqualTo(1);
+ assertImplicitDirectoryFileStatus(statuses[0], fs.makeQualified(statuses[0].getPath()));
+
+ FileStatus[] statuses1 = fs.listStatus(new Path(statuses[0].getPath().toString()));
+ Assertions.assertThat(statuses1.length).isGreaterThanOrEqualTo(1);
+ assertFileFileStatus(statuses1[0], fs.makeQualified(statuses1[0].getPath()));
+ }
+
+ @Test
+ public void testListStatusOnEmptyDirectory() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path emptyDir = new Path("/emptyDir");
+ fs.mkdirs(emptyDir);
+
+ FileStatus[] statuses = fs.listStatus(emptyDir);
+ Assertions.assertThat(statuses.length).isEqualTo(0);
+ }
+
+ @Test
+ public void testContinuationTokenAcrossListStatus() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ fs.listStatus(path);
+
+ ListResponseData listResponseData = fs.getAbfsStore().getClient().listPath(
+ "/testContinuationToken", false, 1, null, getTestTracingContext(fs, true),
+ fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData.getContinuationToken()).isNotNull();
+ Assertions.assertThat(listResponseData.getFileStatusList()).hasSize(1);
+
+ ListResponseData listResponseData1 = fs.getAbfsStore().getClient().listPath(
+ "/testContinuationToken", false, 1, listResponseData.getContinuationToken(), getTestTracingContext(fs, true),
+ fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData1.getContinuationToken()).isNull();
+ Assertions.assertThat(listResponseData1.getFileStatusList()).hasSize(1);
+ }
+
+ @Test
+ public void testInvalidContinuationToken() throws Exception {
+ assumeHnsDisabled();
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testInvalidContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ intercept(AbfsRestOperationException.class,
+ () -> fs.getAbfsStore().getClient().listPath(
+ "/testInvalidContinuationToken", false, 1, "invalidToken",
+ getTestTracingContext(fs, true), fs.getAbfsStore().getUri()));
+ }
+
+ @Test
+ public void testEmptyContinuationToken() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testInvalidContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ ListResponseData listResponseData = fs.getAbfsStore().getClient().listPath(
+ "/testInvalidContinuationToken", false, 1, "",
+ getTestTracingContext(fs, true), fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData.getContinuationToken()).isNotNull();
+ Assertions.assertThat(listResponseData.getFileStatusList()).hasSize(1);
+ }
+
+ private void assertFileFileStatus(final FileStatus fileStatus,
+ final Path qualifiedPath) {
+ Assertions.assertThat(fileStatus.getPath()).isEqualTo(qualifiedPath);
+ Assertions.assertThat(fileStatus.isFile()).isEqualTo(true);
+ Assertions.assertThat(fileStatus.isDirectory()).isEqualTo(false);
+ Assertions.assertThat(fileStatus.getModificationTime()).isNotEqualTo(0);
+ }
+
+ private void assertImplicitDirectoryFileStatus(final FileStatus fileStatus,
+ final Path qualifiedPath) throws Exception {
+ assertDirectoryFileStatus(fileStatus, qualifiedPath);
+ DirectoryStateHelper.isImplicitDirectory(qualifiedPath, getFileSystem(),
+ getTestTracingContext(getFileSystem(), true));
+ Assertions.assertThat(fileStatus.getModificationTime()).isEqualTo(0);
Review Comment:
Assertion statements are missing for all tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014012422
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -285,6 +298,18 @@ private AbfsClient(final URL baseUrl,
metricIdlePeriod);
}
this.abfsMetricUrl = abfsConfiguration.getMetricUri();
+
+ final Class<? extends IdentityTransformerInterface> identityTransformerClass =
Review Comment:
Earlier identity transformation used to happen in Store. Just moved it to client.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014066457
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -404,13 +403,8 @@ public ListResponseData listPath(final String relativePath, final boolean recurs
BlobListResultSchema listResultSchema = getListResultSchemaFromPathStatus(relativePath, pathStatus);
LOG.debug("ListBlob attempted on a file path. Returning file status.");
List<FileStatus> fileStatusList = new ArrayList<>();
- Map<Path, Integer> renamePendingJsonPaths = new HashMap<>();
for (BlobListResultEntrySchema entry : listResultSchema.paths()) {
- if (isRenamePendingJsonPathEntry(entry)) {
- renamePendingJsonPaths.put(entry.path(), entry.contentLength().intValue());
- } else {
- fileStatusList.add(getVersionedFileStatusFromEntry(entry, uri));
- }
+ fileStatusList.add(getVersionedFileStatusFromEntry(entry, uri));
Review Comment:
didn't understand why this change to remove renamePending paths was made
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r1982780815
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1583,26 +1590,40 @@ public Hashtable<String, String> getXMSProperties(AbfsHttpOperation result)
/**
* Parse the XML response body returned by ListBlob API on Blob Endpoint.
- * @param stream InputStream contains the response from server.
- * @return BlobListResultSchema containing the list of entries.
- * @throws IOException if parsing fails.
+ * @param result InputStream contains the response from server.
+ * @param uri to be used for path conversion.
+ * @return {@link ListResponseData}. containing listing response.
+ * @throws AzureBlobFileSystemException if parsing fails.
*/
@Override
- public ListResultSchema parseListPathResults(final InputStream stream) throws IOException {
- if (stream == null) {
- return null;
- }
+ public ListResponseData parseListPathResults(AbfsHttpOperation result, URI uri)
+ throws AzureBlobFileSystemException {
BlobListResultSchema listResultSchema;
- try {
- final SAXParser saxParser = saxParserThreadLocal.get();
- saxParser.reset();
- listResultSchema = new BlobListResultSchema();
- saxParser.parse(stream, new BlobListXmlParser(listResultSchema, getBaseUrl().toString()));
- } catch (SAXException | IOException e) {
- throw new RuntimeException(e);
+ try (InputStream stream = result.getListResultStream()) {
+ if (stream == null) {
+ return null;
+ }
+ try {
+ final SAXParser saxParser = saxParserThreadLocal.get();
+ saxParser.reset();
+ listResultSchema = new BlobListResultSchema();
+ saxParser.parse(stream,
+ new BlobListXmlParser(listResultSchema, getBaseUrl().toString()));
+ result.setListResultSchema(listResultSchema);
+ } catch (SAXException | IOException e) {
+ throw new AbfsDriverException(e);
+ }
+ } catch (IOException e) {
+ LOG.error("Unable to deserialize list results", e);
Review Comment:
Given we have the uri now, should we include that in the error log as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013931246
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -374,63 +380,68 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
requestHeaders);
op.execute(tracingContext);
- // Filter the paths for which no rename redo operation is performed.
- fixAtomicEntriesInListResults(op, tracingContext);
- if (isEmptyListResults(op.getResult()) && is404CheckRequired) {
+ ListResponseData listResponseData = parseListPathResults(op.getResult(), uri);
+ listResponseData.setOp(op);
+
+ // Perform Pending Rename Redo Operation on Atomic Rename Paths.
+ // Crashed HBase log rename recovery can be done by Filesystem.listStatus.
+ if (tracingContext.getOpType() == FSOperationType.LISTSTATUS
+ && op.getResult() != null
+ && op.getResult().getStatusCode() == HTTP_OK) {
+ retryRenameOnAtomicEntriesInListResults(tracingContext,
+ listResponseData.getRenamePendingJsonPaths());
Review Comment:
Yes, this will be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2731787866
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 49s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 50s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 12s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/7/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 12 unchanged - 0 fixed = 14 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 59s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 133m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 70d7863dee1e 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2b13bbd62de448279735425a934b61c869149183 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/7/testReport/ |
| Max. process+thread count | 527 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2732032241
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 45s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 7s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 19s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/8/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 12 unchanged - 0 fixed = 14 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 1m 9s | [/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/8/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 38m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 59s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 133m 27s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Nullcheck of tracingContext at line 388 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsBlobClient.listPath(String, boolean, int, String, TracingContext, URI, boolean) At AbfsBlobClient.java:388 of value previously dereferenced in org.apache.hadoop.fs.azurebfs.services.AbfsBlobClient.listPath(String, boolean, int, String, TracingContext, URI, boolean) At AbfsBlobClient.java:[line 382] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux ca71da58e766 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5d34f65d098fb7cbf8da25c2dd7a81160ef1ca11 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/8/testReport/ |
| Max. process+thread count | 524 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/8/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2697120282
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 49s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 40m 11s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 57s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 1s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 137m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 8621c9c4dac7 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3edab0f5b5ab5d0def2b79d9ac394d4c72001e8e |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/5/testReport/ |
| Max. process+thread count | 524 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013916622
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsAHCHttpOperation.java:
##########
@@ -194,26 +192,14 @@ String getConnResponseMessage() throws IOException {
public void processResponse(final byte[] buffer,
final int offset,
final int length) throws IOException {
- try {
- if (!isPayloadRequest) {
- prepareRequest();
- LOG.debug("Sending request: {}", httpRequestBase);
- httpResponse = executeRequest();
- LOG.debug("Request sent: {}; response {}", httpRequestBase,
- httpResponse);
- }
- parseResponseHeaderAndBody(buffer, offset, length);
- } finally {
Review Comment:
Yes. It was forcing the input stream to close just after response parsing. But with new desing we oved the parsing logic to Clinets. So we need stream to live as long as the ABFSRestOperation object is alive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013944995
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
Review Comment:
can name be shortened
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2753543331
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 8m 19s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 39s | | trunk passed |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 20s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 1s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 14s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 11s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/11/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 12 unchanged - 0 fixed = 13 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 28s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not generate ASF License warnings. |
| | | 80m 19s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux c5625fee2e40 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5a021a00bbcacd9e026eee61ebadf82badadeaa |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/11/testReport/ |
| Max. process+thread count | 683 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013935730
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VersionedFileStatus.java:
##########
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.EtagSource;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * A File status with version info extracted from the etag value returned
+ * in a LIST or HEAD request.
+ * The etag is included in the java serialization.
+ */
+public class VersionedFileStatus extends FileStatus implements EtagSource {
+
+ /**
+ * The superclass is declared serializable; this subclass can also
+ * be serialized.
+ */
+ private static final long serialVersionUID = -2009013240419749458L;
+
+ /**
+ * The etag of an object.
+ * Not-final so that serialization via reflection will preserve the value.
+ */
+ private String version;
+
+ private String encryptionContext;
+
+ public VersionedFileStatus(
+ final String owner, final String group, final FsPermission fsPermission, final boolean hasAcl,
+ final long length, final boolean isdir, final int blockReplication,
+ final long blocksize, final long modificationTime, final Path path,
+ final String version, final String encryptionContext) {
+ super(length, isdir, blockReplication, blocksize, modificationTime, 0,
+ fsPermission,
+ owner,
+ group,
+ null,
+ path,
+ hasAcl, false, false);
+
+ this.version = version;
+ this.encryptionContext = encryptionContext;
+ }
+
+ /** Compare if this object is equal to another object.
+ * @param obj the object to be compared.
+ * @return true if two file status has the same path name; false if not.
+ */
+ @Override
+ public boolean equals(Object obj) {
+ if (!(obj instanceof FileStatus)) {
+ return false;
+ }
+
+ FileStatus other = (FileStatus) obj;
+
+ if (!this.getPath().equals(other.getPath())) {// compare the path
+ return false;
+ }
+
+ if (other instanceof VersionedFileStatus) {
+ return this.version.equals(((VersionedFileStatus) other).version);
+ }
+
+ return true;
+ }
+
+ /**
+ * Returns a hash code value for the object, which is defined as
+ * the hash code of the path name.
+ *
+ * @return a hash code value for the path name and version
+ */
+ @Override
+ public int hashCode() {
+ int hash = getPath().hashCode();
+ hash = 89 * hash + (this.version != null ? this.version.hashCode() : 0);
+ return hash;
+ }
+
+ /**
+ * Returns the version of this FileStatus
+ *
+ * @return a string value for the FileStatus version
+ */
+ public String getVersion() {
+ return this.version;
+ }
+
+ @Override
+ public String getEtag() {
Review Comment:
Taken
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java:
##########
@@ -668,6 +660,10 @@ protected boolean isConnectionDisconnectedOnError() {
return connectionDisconnectedOnError;
}
+ protected void setListResultSchema(final ListResultSchema listResultSchema) {
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013937894
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
+ try {
+ primaryUserGroup = UserGroupInformation.getCurrentUser().getPrimaryGroupName();
+ } catch (IOException ex) {
+ LOG.error("Failed to get primary group for {}, using user name as primary group name",
+ getPrimaryUser());
+ primaryUserGroup = getPrimaryUser();
+ }
+ } else {
+ //Provide a default group name
+ primaryUserGroup = getPrimaryUser();
+ }
+ return primaryUserGroup;
+ }
+
+ private String getPrimaryUser() throws AzureBlobFileSystemException {
+ try {
+ return UserGroupInformation.getCurrentUser().getUserName();
+ } catch (IOException ex) {
+ throw new AbfsDriverException(ex);
+ }
+ }
+
+ /**
+ * Creates a VersionedFileStatus object from the ListResultEntrySchema.
+ * @param entry ListResultEntrySchema object.
+ * @param uri to be used for the path conversion.
+ * @return VersionedFileStatus object.
+ * @throws AzureBlobFileSystemException if transformation fails.
+ */
+ protected VersionedFileStatus getVersionedFileStatusFromEntry(
+ ListResultEntrySchema entry, URI uri) throws AzureBlobFileSystemException {
+ long blockSize = abfsConfiguration.getAzureBlockSize();
+ final String owner, group;
+ try{
+ if (identityTransformer != null) {
+ owner = identityTransformer.transformIdentityForGetRequest(
+ entry.owner(), true, getPrimaryUser());
+ group = identityTransformer.transformIdentityForGetRequest(
+ entry.group(), false, getPrimaryUserGroup());
+ } else {
+ owner = null;
+ group = null;
+ }
+ } catch (IOException ex) {
+ LOG.error("Failed to get owner/group for path {}", entry.name(), ex);
+ throw new AbfsDriverException(ex);
+ }
+ final String encryptionContext = entry.getXMsEncryptionContext();
Review Comment:
It will always be non-null value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013944402
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
Review Comment:
javadocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 merged PR #7421:
URL: https://github.com/apache/hadoop/pull/7421
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013934055
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VersionedFileStatus.java:
##########
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.EtagSource;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * A File status with version info extracted from the etag value returned
+ * in a LIST or HEAD request.
+ * The etag is included in the java serialization.
+ */
+public class VersionedFileStatus extends FileStatus implements EtagSource {
+
+ /**
+ * The superclass is declared serializable; this subclass can also
+ * be serialized.
+ */
+ private static final long serialVersionUID = -2009013240419749458L;
+
+ /**
+ * The etag of an object.
+ * Not-final so that serialization via reflection will preserve the value.
+ */
+ private String version;
+
+ private String encryptionContext;
+
+ public VersionedFileStatus(
+ final String owner, final String group, final FsPermission fsPermission, final boolean hasAcl,
+ final long length, final boolean isdir, final int blockReplication,
+ final long blocksize, final long modificationTime, final Path path,
+ final String version, final String encryptionContext) {
+ super(length, isdir, blockReplication, blocksize, modificationTime, 0,
+ fsPermission,
+ owner,
+ group,
+ null,
+ path,
+ hasAcl, false, false);
+
+ this.version = version;
+ this.encryptionContext = encryptionContext;
+ }
+
+ /** Compare if this object is equal to another object.
+ * @param obj the object to be compared.
+ * @return true if two file status has the same path name; false if not.
+ */
+ @Override
+ public boolean equals(Object obj) {
+ if (!(obj instanceof FileStatus)) {
+ return false;
+ }
+
+ FileStatus other = (FileStatus) obj;
+
+ if (!this.getPath().equals(other.getPath())) {// compare the path
+ return false;
+ }
+
+ if (other instanceof VersionedFileStatus) {
+ return this.version.equals(((VersionedFileStatus) other).version);
+ }
+
+ return true;
+ }
+
+ /**
+ * Returns a hash code value for the object, which is defined as
+ * the hash code of the path name.
+ *
+ * @return a hash code value for the path name and version
+ */
+ @Override
+ public int hashCode() {
+ int hash = getPath().hashCode();
+ hash = 89 * hash + (this.version != null ? this.version.hashCode() : 0);
Review Comment:
Not very sure, its a day 0 code just moved it out from Store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013937192
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java:
##########
@@ -1464,20 +1472,41 @@ public Hashtable<String, String> getXMSProperties(AbfsHttpOperation result)
/**
* Parse the list file response from DFS ListPath API in Json format
- * @param stream InputStream contains the list results.
- * @throws IOException if parsing fails.
+ * @param result InputStream contains the list results.
+ * @param uri to be used for path conversion.
+ * @return {@link ListResponseData}. containing listing response.
+ * @throws AzureBlobFileSystemException if parsing fails.
*/
@Override
- public ListResultSchema parseListPathResults(final InputStream stream) throws IOException {
- DfsListResultSchema listResultSchema;
- try {
- final ObjectMapper objectMapper = new ObjectMapper();
- listResultSchema = objectMapper.readValue(stream, DfsListResultSchema.class);
+ public ListResponseData parseListPathResults(AbfsHttpOperation result, URI uri) throws AzureBlobFileSystemException {
+ try (InputStream listResultInputStream = result.getListResultStream()) {
+ DfsListResultSchema listResultSchema;
+ try {
+ final ObjectMapper objectMapper = new ObjectMapper();
+ listResultSchema = objectMapper.readValue(listResultInputStream,
+ DfsListResultSchema.class);
+ result.setListResultSchema(listResultSchema);
+ LOG.debug("ListPath listed {} paths with {} as continuation token",
+ listResultSchema.paths().size(),
+ getContinuationFromResponse(result));
+ } catch (IOException ex) {
+ throw new AbfsDriverException(ex);
Review Comment:
There is catch outside which will log error for this exception as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013951578
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java:
##########
@@ -1464,20 +1472,41 @@ public Hashtable<String, String> getXMSProperties(AbfsHttpOperation result)
/**
* Parse the list file response from DFS ListPath API in Json format
- * @param stream InputStream contains the list results.
- * @throws IOException if parsing fails.
+ * @param result InputStream contains the list results.
+ * @param uri to be used for path conversion.
+ * @return {@link ListResponseData}. containing listing response.
+ * @throws AzureBlobFileSystemException if parsing fails.
*/
@Override
- public ListResultSchema parseListPathResults(final InputStream stream) throws IOException {
- DfsListResultSchema listResultSchema;
- try {
- final ObjectMapper objectMapper = new ObjectMapper();
- listResultSchema = objectMapper.readValue(stream, DfsListResultSchema.class);
+ public ListResponseData parseListPathResults(AbfsHttpOperation result, URI uri) throws AzureBlobFileSystemException {
+ try (InputStream listResultInputStream = result.getListResultStream()) {
+ DfsListResultSchema listResultSchema;
+ try {
+ final ObjectMapper objectMapper = new ObjectMapper();
+ listResultSchema = objectMapper.readValue(listResultInputStream,
+ DfsListResultSchema.class);
+ result.setListResultSchema(listResultSchema);
+ LOG.debug("ListPath listed {} paths with {} as continuation token",
+ listResultSchema.paths().size(),
+ getContinuationFromResponse(result));
+ } catch (IOException ex) {
+ throw new AbfsDriverException(ex);
+ }
+
+ List<FileStatus> fileStatuses = new ArrayList<>();
+ for (DfsListResultEntrySchema entry : listResultSchema.paths()) {
+ fileStatuses.add(getVersionedFileStatusFromEntry(entry, uri));
+ }
+ ListResponseData listResponseData = new ListResponseData();
+ listResponseData.setFileStatusList(fileStatuses);
+ listResponseData.setRenamePendingJsonPaths(null);
+ listResponseData.setContinuationToken(
+ getContinuationFromResponse(result));
+ return listResponseData;
} catch (IOException ex) {
LOG.error("Unable to deserialize list results", ex);
Review Comment:
Add the uri as well in the exception message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014066457
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -404,13 +403,8 @@ public ListResponseData listPath(final String relativePath, final boolean recurs
BlobListResultSchema listResultSchema = getListResultSchemaFromPathStatus(relativePath, pathStatus);
LOG.debug("ListBlob attempted on a file path. Returning file status.");
List<FileStatus> fileStatusList = new ArrayList<>();
- Map<Path, Integer> renamePendingJsonPaths = new HashMap<>();
for (BlobListResultEntrySchema entry : listResultSchema.paths()) {
- if (isRenamePendingJsonPathEntry(entry)) {
- renamePendingJsonPaths.put(entry.path(), entry.contentLength().intValue());
- } else {
- fileStatusList.add(getVersionedFileStatusFromEntry(entry, uri));
- }
+ fileStatusList.add(getVersionedFileStatusFromEntry(entry, uri));
Review Comment:
didn't understand why this change to remove renamePending paths was removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013943423
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -285,6 +298,18 @@ private AbfsClient(final URL baseUrl,
metricIdlePeriod);
}
this.abfsMetricUrl = abfsConfiguration.getMetricUri();
+
+ final Class<? extends IdentityTransformerInterface> identityTransformerClass =
Review Comment:
what is this change for identity transformer related to ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014026084
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestListActionTaker.java:
##########
@@ -132,10 +136,10 @@ protected void addPaths(final List<Path> paths,
occurrences[0]++;
Assertions.assertThat((int) answer.getArgument(2))
Review Comment:
assertion statement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2757494851
============================================================
HNS-OAuth-DFS
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 796, Failures: 0, Errors: 0, Skipped: 162
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 23
============================================================
HNS-SharedKey-DFS
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 799, Failures: 0, Errors: 0, Skipped: 131
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 10
============================================================
NonHNS-SharedKey-DFS
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 638, Failures: 0, Errors: 0, Skipped: 214
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 11
============================================================
AppendBlob-HNS-OAuth-DFS
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 796, Failures: 0, Errors: 0, Skipped: 169
[WARNING] Tests run: 131, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 23
============================================================
NonHNS-SharedKey-Blob
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 641, Failures: 0, Errors: 0, Skipped: 144
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 11
============================================================
NonHNS-OAuth-DFS
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 635, Failures: 0, Errors: 0, Skipped: 215
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24
============================================================
NonHNS-OAuth-Blob
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 638, Failures: 0, Errors: 0, Skipped: 145
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24
============================================================
AppendBlob-NonHNS-OAuth-Blob
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 636, Failures: 0, Errors: 0, Skipped: 163
[WARNING] Tests run: 131, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24
============================================================
HNS-Oauth-DFS-IngressBlob
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 670, Failures: 0, Errors: 0, Skipped: 165
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 23
============================================================
NonHNS-OAuth-DFS-IngressBlob
============================================================
[WARNING] Tests run: 174, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 635, Failures: 0, Errors: 0, Skipped: 213
[WARNING] Tests run: 154, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 272, Failures: 0, Errors: 0, Skipped: 24
Branch: HADOOP-19234_followup, Commit: 972889915dbb4f9c53162838f56f6bbaeda2431e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013976701
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -1780,19 +1782,19 @@ private Configuration createConfig(String producerQueueSize, String consumerMaxL
private void validateRename(AzureBlobFileSystem fs, Path src, Path dst,
boolean isSrcExist, boolean isDstExist, boolean isJsonExist)
throws IOException {
- Assertions.assertThat(fs.exists(dst))
- .describedAs("Renamed Destination directory should exist.")
- .isEqualTo(isDstExist);
Assertions.assertThat(fs.exists(new Path(src.getParent(), src.getName() + SUFFIX)))
.describedAs("Renamed Pending Json file should exist.")
.isEqualTo(isJsonExist);
Assertions.assertThat(fs.exists(src))
- .describedAs("Renamed Destination directory should exist.")
+ .describedAs("Renamed Source directory should exist.")
.isEqualTo(isSrcExist);
+ Assertions.assertThat(fs.exists(dst))
+ .describedAs("Renamed Destination directory should exist.")
+ .isEqualTo(isDstExist);
}
/**
- * Test the renaming of a directory with different parallelism configurations.
+ * Test the renaming of a directory with different parallelism configurations.c
Review Comment:
typo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r1982791170
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1903,39 +1916,57 @@ private List<AbfsHttpHeader> getMetadataHeadersList(final Hashtable<String, Stri
* This is to handle duplicate listing entries returned by Blob Endpoint for
* implicit paths that also has a marker file created for them.
* This will retain entry corresponding to marker file and remove the BlobPrefix entry.
+ * This will also filter out all the rename pending json files in listing output.
* @param listResultSchema List of entries returned by Blob Endpoint.
+ * @param uri URI to be used for path conversion.
* @return List of entries after removing duplicates.
*/
- private BlobListResultSchema removeDuplicateEntries(BlobListResultSchema listResultSchema) {
- List<BlobListResultEntrySchema> uniqueEntries = new ArrayList<>();
+ private ListResponseData filterDuplicateEntriesAndRenamePendingFiles(
+ BlobListResultSchema listResultSchema, URI uri) throws IOException {
+ List<FileStatus> fileStatuses = new ArrayList<>();
+ Map<Path, Integer> renamePendingJsonPaths = new HashMap<>();
TreeMap<String, BlobListResultEntrySchema> nameToEntryMap = new TreeMap<>();
for (BlobListResultEntrySchema entry : listResultSchema.paths()) {
if (StringUtils.isNotEmpty(entry.eTag())) {
// This is a blob entry. It is either a file or a marker blob.
// In both cases we will add this.
nameToEntryMap.put(entry.name(), entry);
+ fileStatuses.add(getVersionedFileStatusFromEntry(entry, uri));
+
+ if (isRenamePendingJsonPathEntry(entry)) {
+ renamePendingJsonPaths.put(entry.path(), entry.contentLength().intValue());
+ }
} else {
// This is a BlobPrefix entry. It is a directory with file inside
// This might have already been added as a marker blob.
if (!nameToEntryMap.containsKey(entry.name())) {
nameToEntryMap.put(entry.name(), entry);
+ fileStatuses.add(getVersionedFileStatusFromEntry(entry, uri));
}
}
}
- uniqueEntries.addAll(nameToEntryMap.values());
- listResultSchema.withPaths(uniqueEntries);
- return listResultSchema;
+ ListResponseData listResponseData = new ListResponseData();
+ listResponseData.setFileStatusList(fileStatuses);
+ listResponseData.setRenamePendingJsonPaths(renamePendingJsonPaths);
+ listResponseData.setContinuationToken(listResultSchema.getNextMarker());
+ return listResponseData;
+ }
+
+ private boolean isRenamePendingJsonPathEntry(BlobListResultEntrySchema entry) {
Review Comment:
missing javadocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013589793
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1277,57 +1275,14 @@ public String listStatus(final Path path, final String startFrom,
do {
try (AbfsPerfInfo perfInfo = startTracking("listStatus", "listPath")) {
- AbfsRestOperation op = listingClient.listPath(relativePath, false,
- abfsConfiguration.getListMaxResults(), continuation,
- tracingContext);
+ ListResponseData listResponseData = listingClient.listPath(relativePath,
Review Comment:
Should we perform a null check on listResponseData in this case?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsAHCHttpOperation.java:
##########
@@ -194,26 +192,14 @@ String getConnResponseMessage() throws IOException {
public void processResponse(final byte[] buffer,
final int offset,
final int length) throws IOException {
- try {
- if (!isPayloadRequest) {
- prepareRequest();
- LOG.debug("Sending request: {}", httpRequestBase);
- httpResponse = executeRequest();
- LOG.debug("Request sent: {}; response {}", httpRequestBase,
- httpResponse);
- }
- parseResponseHeaderAndBody(buffer, offset, length);
- } finally {
Review Comment:
Any reason for removing this part of code?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
+ try {
+ primaryUserGroup = UserGroupInformation.getCurrentUser().getPrimaryGroupName();
+ } catch (IOException ex) {
+ LOG.error("Failed to get primary group for {}, using user name as primary group name",
+ getPrimaryUser());
+ primaryUserGroup = getPrimaryUser();
+ }
+ } else {
+ //Provide a default group name
+ primaryUserGroup = getPrimaryUser();
+ }
+ return primaryUserGroup;
+ }
+
+ private String getPrimaryUser() throws AzureBlobFileSystemException {
Review Comment:
Java doc missing
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VersionedFileStatus.java:
##########
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.EtagSource;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * A File status with version info extracted from the etag value returned
+ * in a LIST or HEAD request.
+ * The etag is included in the java serialization.
+ */
+public class VersionedFileStatus extends FileStatus implements EtagSource {
+
+ /**
+ * The superclass is declared serializable; this subclass can also
+ * be serialized.
+ */
+ private static final long serialVersionUID = -2009013240419749458L;
+
+ /**
+ * The etag of an object.
+ * Not-final so that serialization via reflection will preserve the value.
+ */
+ private String version;
+
+ private String encryptionContext;
+
+ public VersionedFileStatus(
+ final String owner, final String group, final FsPermission fsPermission, final boolean hasAcl,
+ final long length, final boolean isdir, final int blockReplication,
+ final long blocksize, final long modificationTime, final Path path,
+ final String version, final String encryptionContext) {
+ super(length, isdir, blockReplication, blocksize, modificationTime, 0,
+ fsPermission,
+ owner,
+ group,
+ null,
+ path,
+ hasAcl, false, false);
+
+ this.version = version;
+ this.encryptionContext = encryptionContext;
+ }
+
+ /** Compare if this object is equal to another object.
+ * @param obj the object to be compared.
+ * @return true if two file status has the same path name; false if not.
+ */
+ @Override
+ public boolean equals(Object obj) {
+ if (!(obj instanceof FileStatus)) {
+ return false;
+ }
+
+ FileStatus other = (FileStatus) obj;
+
+ if (!this.getPath().equals(other.getPath())) {// compare the path
+ return false;
+ }
+
+ if (other instanceof VersionedFileStatus) {
+ return this.version.equals(((VersionedFileStatus) other).version);
+ }
+
+ return true;
+ }
+
+ /**
+ * Returns a hash code value for the object, which is defined as
+ * the hash code of the path name.
+ *
+ * @return a hash code value for the path name and version
+ */
+ @Override
+ public int hashCode() {
+ int hash = getPath().hashCode();
+ hash = 89 * hash + (this.version != null ? this.version.hashCode() : 0);
Review Comment:
What is the logic behind multiplying hash with 89?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsDfsClient.java:
##########
@@ -1464,20 +1472,41 @@ public Hashtable<String, String> getXMSProperties(AbfsHttpOperation result)
/**
* Parse the list file response from DFS ListPath API in Json format
- * @param stream InputStream contains the list results.
- * @throws IOException if parsing fails.
+ * @param result InputStream contains the list results.
+ * @param uri to be used for path conversion.
+ * @return {@link ListResponseData}. containing listing response.
+ * @throws AzureBlobFileSystemException if parsing fails.
*/
@Override
- public ListResultSchema parseListPathResults(final InputStream stream) throws IOException {
- DfsListResultSchema listResultSchema;
- try {
- final ObjectMapper objectMapper = new ObjectMapper();
- listResultSchema = objectMapper.readValue(stream, DfsListResultSchema.class);
+ public ListResponseData parseListPathResults(AbfsHttpOperation result, URI uri) throws AzureBlobFileSystemException {
+ try (InputStream listResultInputStream = result.getListResultStream()) {
+ DfsListResultSchema listResultSchema;
+ try {
+ final ObjectMapper objectMapper = new ObjectMapper();
+ listResultSchema = objectMapper.readValue(listResultInputStream,
+ DfsListResultSchema.class);
+ result.setListResultSchema(listResultSchema);
+ LOG.debug("ListPath listed {} paths with {} as continuation token",
+ listResultSchema.paths().size(),
+ getContinuationFromResponse(result));
+ } catch (IOException ex) {
+ throw new AbfsDriverException(ex);
Review Comment:
Please add some error message along with exception.
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -1780,19 +1782,19 @@ private Configuration createConfig(String producerQueueSize, String consumerMaxL
private void validateRename(AzureBlobFileSystem fs, Path src, Path dst,
boolean isSrcExist, boolean isDstExist, boolean isJsonExist)
throws IOException {
- Assertions.assertThat(fs.exists(dst))
- .describedAs("Renamed Destination directory should exist.")
- .isEqualTo(isDstExist);
Assertions.assertThat(fs.exists(new Path(src.getParent(), src.getName() + SUFFIX)))
.describedAs("Renamed Pending Json file should exist.")
.isEqualTo(isJsonExist);
Assertions.assertThat(fs.exists(src))
- .describedAs("Renamed Destination directory should exist.")
+ .describedAs("Renamed Source directory should exist.")
Review Comment:
Could you please correct it to `Renamed Source directory should not exist.`
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
Review Comment:
We can simplify this method
```
private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
try {
return UserGroupInformation.getCurrentUser().getPrimaryGroupName();
} catch (IOException ex) {
LOG.error("Failed to get primary group for {}, using user name as primary group name",
getPrimaryUser());
}
}
return getPrimaryUser();
}
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1277,57 +1275,14 @@ public String listStatus(final Path path, final String startFrom,
do {
try (AbfsPerfInfo perfInfo = startTracking("listStatus", "listPath")) {
- AbfsRestOperation op = listingClient.listPath(relativePath, false,
- abfsConfiguration.getListMaxResults(), continuation,
- tracingContext);
+ ListResponseData listResponseData = listingClient.listPath(relativePath,
+ false, abfsConfiguration.getListMaxResults(), continuation,
+ tracingContext, this.uri);
+ AbfsRestOperation op = listResponseData.getOp();
perfInfo.registerResult(op.getResult());
- continuation = listingClient.getContinuationFromResponse(op.getResult());
- ListResultSchema retrievedSchema = op.getResult().getListResultSchema();
- if (retrievedSchema == null) {
- throw new AbfsRestOperationException(
- AzureServiceErrorCode.PATH_NOT_FOUND.getStatusCode(),
- AzureServiceErrorCode.PATH_NOT_FOUND.getErrorCode(),
- "listStatusAsync path not found",
- null, op.getResult());
- }
-
- long blockSize = abfsConfiguration.getAzureBlockSize();
-
- for (ListResultEntrySchema entry : retrievedSchema.paths()) {
- final String owner = identityTransformer.transformIdentityForGetRequest(entry.owner(), true, userName);
- final String group = identityTransformer.transformIdentityForGetRequest(entry.group(), false, primaryUserGroup);
- final String encryptionContext = entry.getXMsEncryptionContext();
- final FsPermission fsPermission = entry.permissions() == null
- ? new AbfsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL)
- : AbfsPermission.valueOf(entry.permissions());
- final boolean hasAcl = AbfsPermission.isExtendedAcl(entry.permissions());
-
- long lastModifiedMillis = 0;
- long contentLength = entry.contentLength() == null ? 0 : entry.contentLength();
- boolean isDirectory = entry.isDirectory() == null ? false : entry.isDirectory();
- if (entry.lastModified() != null && !entry.lastModified().isEmpty()) {
- lastModifiedMillis = DateTimeUtils.parseLastModifiedTime(
- entry.lastModified());
- }
-
- Path entryPath = new Path(File.separator + entry.name());
- entryPath = entryPath.makeQualified(this.uri, entryPath);
-
- fileStatuses.add(
- new VersionedFileStatus(
- owner,
- group,
- fsPermission,
- hasAcl,
- contentLength,
- isDirectory,
- 1,
- blockSize,
- lastModifiedMillis,
- entryPath,
- entry.eTag(),
- encryptionContext));
- }
+ continuation = listResponseData.getContinuationToken();
+ List<FileStatus> fileStatusListInCurrItr = listResponseData.getFileStatusList();
+ fileStatuses.addAll(fileStatusListInCurrItr);
Review Comment:
Before adding it to fileStatuses, shouldn't we check if fileStatusListInCurrItr is null or empty?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.services;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation;
+
+/**
+ * This class is used to hold the response data for list operations.
+ * It contains a list of FileStatus objects, a map of rename pending JSON paths,
+ * continuation token and the executed REST operation.
+ */
+public class ListResponseData {
+
+ private List<FileStatus> fileStatusList;
+ private Map<Path, Integer> renamePendingJsonPaths;
Review Comment:
Since content Length is long, we should keep it Map<Path, Long>.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -285,6 +298,18 @@ private AbfsClient(final URL baseUrl,
metricIdlePeriod);
}
this.abfsMetricUrl = abfsConfiguration.getMetricUri();
+
+ final Class<? extends IdentityTransformerInterface> identityTransformerClass =
+ abfsConfiguration.getRawConfiguration().getClass(FS_AZURE_IDENTITY_TRANSFORM_CLASS, IdentityTransformer.class,
+ IdentityTransformerInterface.class);
+ try {
+ this.identityTransformer =
+ identityTransformerClass.getConstructor(Configuration.class).newInstance(abfsConfiguration.getRawConfiguration());
+ } catch (IllegalAccessException | InstantiationException | IllegalArgumentException
+ | InvocationTargetException | NoSuchMethodException e) {
+ throw new IOException(e);
Review Comment:
We can add some message as well along with exception for better understanding.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
+ try {
+ primaryUserGroup = UserGroupInformation.getCurrentUser().getPrimaryGroupName();
+ } catch (IOException ex) {
+ LOG.error("Failed to get primary group for {}, using user name as primary group name",
+ getPrimaryUser());
+ primaryUserGroup = getPrimaryUser();
+ }
+ } else {
+ //Provide a default group name
+ primaryUserGroup = getPrimaryUser();
+ }
+ return primaryUserGroup;
+ }
+
+ private String getPrimaryUser() throws AzureBlobFileSystemException {
+ try {
+ return UserGroupInformation.getCurrentUser().getUserName();
+ } catch (IOException ex) {
+ throw new AbfsDriverException(ex);
+ }
+ }
+
+ /**
+ * Creates a VersionedFileStatus object from the ListResultEntrySchema.
+ * @param entry ListResultEntrySchema object.
+ * @param uri to be used for the path conversion.
+ * @return VersionedFileStatus object.
+ * @throws AzureBlobFileSystemException if transformation fails.
+ */
+ protected VersionedFileStatus getVersionedFileStatusFromEntry(
+ ListResultEntrySchema entry, URI uri) throws AzureBlobFileSystemException {
+ long blockSize = abfsConfiguration.getAzureBlockSize();
+ final String owner, group;
+ try{
+ if (identityTransformer != null) {
+ owner = identityTransformer.transformIdentityForGetRequest(
+ entry.owner(), true, getPrimaryUser());
+ group = identityTransformer.transformIdentityForGetRequest(
+ entry.group(), false, getPrimaryUserGroup());
+ } else {
+ owner = null;
Review Comment:
The default value for owner and group is null. Do we need to explicitly assign it to null here?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
Review Comment:
Java doc for this method
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1753,4 +1772,87 @@ protected AbfsRestOperation getSuccessOp(final AbfsRestOperationType operationTy
successOp.hardSetResult(HttpURLConnection.HTTP_OK);
return successOp;
}
+
+ private String getPrimaryUserGroup() throws AzureBlobFileSystemException {
+ String primaryUserGroup;
+ if (!getAbfsConfiguration().getSkipUserGroupMetadataDuringInitialization()) {
+ try {
+ primaryUserGroup = UserGroupInformation.getCurrentUser().getPrimaryGroupName();
+ } catch (IOException ex) {
+ LOG.error("Failed to get primary group for {}, using user name as primary group name",
+ getPrimaryUser());
+ primaryUserGroup = getPrimaryUser();
+ }
+ } else {
+ //Provide a default group name
+ primaryUserGroup = getPrimaryUser();
+ }
+ return primaryUserGroup;
+ }
+
+ private String getPrimaryUser() throws AzureBlobFileSystemException {
+ try {
+ return UserGroupInformation.getCurrentUser().getUserName();
+ } catch (IOException ex) {
+ throw new AbfsDriverException(ex);
+ }
+ }
+
+ /**
+ * Creates a VersionedFileStatus object from the ListResultEntrySchema.
+ * @param entry ListResultEntrySchema object.
+ * @param uri to be used for the path conversion.
+ * @return VersionedFileStatus object.
+ * @throws AzureBlobFileSystemException if transformation fails.
+ */
+ protected VersionedFileStatus getVersionedFileStatusFromEntry(
+ ListResultEntrySchema entry, URI uri) throws AzureBlobFileSystemException {
+ long blockSize = abfsConfiguration.getAzureBlockSize();
+ final String owner, group;
+ try{
+ if (identityTransformer != null) {
+ owner = identityTransformer.transformIdentityForGetRequest(
+ entry.owner(), true, getPrimaryUser());
+ group = identityTransformer.transformIdentityForGetRequest(
+ entry.group(), false, getPrimaryUserGroup());
+ } else {
+ owner = null;
+ group = null;
+ }
+ } catch (IOException ex) {
+ LOG.error("Failed to get owner/group for path {}", entry.name(), ex);
+ throw new AbfsDriverException(ex);
+ }
+ final String encryptionContext = entry.getXMsEncryptionContext();
Review Comment:
Should we perform a null check on the entry before this, or will the entry always have a non-null value?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java:
##########
@@ -668,6 +660,10 @@ protected boolean isConnectionDisconnectedOnError() {
return connectionDisconnectedOnError;
}
+ protected void setListResultSchema(final ListResultSchema listResultSchema) {
Review Comment:
Java doc missing
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -374,63 +380,68 @@ public AbfsRestOperation listPath(final String relativePath, final boolean recur
requestHeaders);
op.execute(tracingContext);
- // Filter the paths for which no rename redo operation is performed.
- fixAtomicEntriesInListResults(op, tracingContext);
- if (isEmptyListResults(op.getResult()) && is404CheckRequired) {
+ ListResponseData listResponseData = parseListPathResults(op.getResult(), uri);
+ listResponseData.setOp(op);
+
+ // Perform Pending Rename Redo Operation on Atomic Rename Paths.
+ // Crashed HBase log rename recovery can be done by Filesystem.listStatus.
+ if (tracingContext.getOpType() == FSOperationType.LISTSTATUS
+ && op.getResult() != null
+ && op.getResult().getStatusCode() == HTTP_OK) {
+ retryRenameOnAtomicEntriesInListResults(tracingContext,
+ listResponseData.getRenamePendingJsonPaths());
Review Comment:
We are calculating renamePendingJsonPath in this method after this line. How listResponseData.getRenamePendingJsonPaths() will work here is not clear.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VersionedFileStatus.java:
##########
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.EtagSource;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * A File status with version info extracted from the etag value returned
+ * in a LIST or HEAD request.
+ * The etag is included in the java serialization.
+ */
+public class VersionedFileStatus extends FileStatus implements EtagSource {
+
+ /**
+ * The superclass is declared serializable; this subclass can also
+ * be serialized.
+ */
+ private static final long serialVersionUID = -2009013240419749458L;
+
+ /**
+ * The etag of an object.
+ * Not-final so that serialization via reflection will preserve the value.
+ */
+ private String version;
+
+ private String encryptionContext;
+
+ public VersionedFileStatus(
+ final String owner, final String group, final FsPermission fsPermission, final boolean hasAcl,
+ final long length, final boolean isdir, final int blockReplication,
+ final long blocksize, final long modificationTime, final Path path,
+ final String version, final String encryptionContext) {
+ super(length, isdir, blockReplication, blocksize, modificationTime, 0,
+ fsPermission,
+ owner,
+ group,
+ null,
+ path,
+ hasAcl, false, false);
+
+ this.version = version;
+ this.encryptionContext = encryptionContext;
+ }
+
+ /** Compare if this object is equal to another object.
+ * @param obj the object to be compared.
+ * @return true if two file status has the same path name; false if not.
+ */
+ @Override
+ public boolean equals(Object obj) {
+ if (!(obj instanceof FileStatus)) {
+ return false;
+ }
+
+ FileStatus other = (FileStatus) obj;
+
+ if (!this.getPath().equals(other.getPath())) {// compare the path
+ return false;
+ }
+
+ if (other instanceof VersionedFileStatus) {
+ return this.version.equals(((VersionedFileStatus) other).version);
+ }
+
+ return true;
+ }
+
+ /**
+ * Returns a hash code value for the object, which is defined as
+ * the hash code of the path name.
+ *
+ * @return a hash code value for the path name and version
+ */
+ @Override
+ public int hashCode() {
+ int hash = getPath().hashCode();
+ hash = 89 * hash + (this.version != null ? this.version.hashCode() : 0);
+ return hash;
+ }
+
+ /**
+ * Returns the version of this FileStatus
+ *
+ * @return a string value for the FileStatus version
+ */
+ public String getVersion() {
+ return this.version;
+ }
+
+ @Override
+ public String getEtag() {
Review Comment:
Java doc missing
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.services;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation;
+
+/**
+ * This class is used to hold the response data for list operations.
+ * It contains a list of FileStatus objects, a map of rename pending JSON paths,
+ * continuation token and the executed REST operation.
+ */
+public class ListResponseData {
+
+ private List<FileStatus> fileStatusList;
+ private Map<Path, Integer> renamePendingJsonPaths;
Review Comment:
Keeping it as an Integer is also fine. Since the JSON file will be small, it's okay to keep it as an int.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.services;
Review Comment:
I think we should either create a new package and keep this file there or move it to the service package since there are many similar files present there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013835275
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VersionedFileStatus.java:
##########
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.EtagSource;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * A File status with version info extracted from the etag value returned
+ * in a LIST or HEAD request.
+ * The etag is included in the java serialization.
+ */
+public class VersionedFileStatus extends FileStatus implements EtagSource {
Review Comment:
javadocs missing for some functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013911075
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.services;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation;
+
+/**
+ * This class is used to hold the response data for list operations.
+ * It contains a list of FileStatus objects, a map of rename pending JSON paths,
+ * continuation token and the executed REST operation.
+ */
+public class ListResponseData {
+
+ private List<FileStatus> fileStatusList;
+ private Map<Path, Integer> renamePendingJsonPaths;
Review Comment:
I think its a good suggestion to keep it long only. Everywhere we parse content lenght as long only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013955480
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java:
##########
@@ -383,7 +388,8 @@ final void parseResponse(final byte[] buffer,
// consume the input stream to release resources
int totalBytesRead = 0;
- try (InputStream stream = getContentInputStream()) {
+ try {
Review Comment:
try with resources was ensuring stream is closed, is that not needed now ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014009551
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResponseData.java:
##########
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.contracts.services;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation;
+
+/**
+ * This class is used to hold the response data for list operations.
+ * It contains a list of FileStatus objects, a map of rename pending JSON paths,
+ * continuation token and the executed REST operation.
+ */
+public class ListResponseData {
+
+ private List<FileStatus> fileStatusList;
+ private Map<Path, Integer> renamePendingJsonPaths;
Review Comment:
Rename Atomicity anyway accepts an integer only, that's why kept as integer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013968726
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemListStatus.java:
##########
@@ -343,4 +319,174 @@ public void testRenameTrailingPeriodFile() throws IOException {
assertTrue("Attempt to create file that ended with a dot should"
+ " throw IllegalArgumentException", exceptionThrown);
}
+
+
+
+ /**
+ * Test to verify that listStatus returns the correct file status all types
+ * of paths viz. implicit, explicit, file.
+ * @throws Exception if there is an error or test assertions fails.
+ */
+ @Test
+ public void testListStatusWithImplicitExplicitChildren() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ fs.setWorkingDirectory(new Path(ROOT_PATH));
+ Path root = new Path(ROOT_PATH);
+
+ // Create an implicit directory under root
+ Path dir = new Path("a");
+ Path fileInsideDir = new Path("a/file");
+ createAzCopyFolder(dir);
+
+ // Assert that implicit directory is returned
+ FileStatus[] fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(1);
+ assertImplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+
+ // Create a marker blob for the directory.
+ fs.create(fileInsideDir);
+
+ // Assert that only one entry of explicit directory is returned
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(1);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+
+ // Create a file under root
+ Path file1 = new Path("b");
+ fs.create(file1);
+
+ // Assert that two entries are returned in alphabetic order.
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(2);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+ assertFileFileStatus(fileStatuses[1], fs.makeQualified(file1));
+
+ // Create another implicit directory under root.
+ Path dir2 = new Path("c");
+ createAzCopyFolder(dir2);
+
+ // Assert that three entries are returned in alphabetic order.
+ fileStatuses = fs.listStatus(root);
+ Assertions.assertThat(fileStatuses.length).isEqualTo(3);
+ assertExplicitDirectoryFileStatus(fileStatuses[0], fs.makeQualified(dir));
+ assertFileFileStatus(fileStatuses[1], fs.makeQualified(file1));
+ assertImplicitDirectoryFileStatus(fileStatuses[2], fs.makeQualified(dir2));
+ }
+
+ /**
+ * Test to verify that listStatus returns the correct file status when called on an implicit path
+ * @throws Exception if there is an error or test assertions fails.
+ */
+ @Test
+ public void testListStatusOnImplicitDirectoryPath() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path implicitPath = new Path("/implicitDir");
+ createAzCopyFolder(implicitPath);
+
+ FileStatus[] statuses = fs.listStatus(implicitPath);
+ Assertions.assertThat(statuses.length).isGreaterThanOrEqualTo(1);
+ assertImplicitDirectoryFileStatus(statuses[0], fs.makeQualified(statuses[0].getPath()));
+
+ FileStatus[] statuses1 = fs.listStatus(new Path(statuses[0].getPath().toString()));
+ Assertions.assertThat(statuses1.length).isGreaterThanOrEqualTo(1);
+ assertFileFileStatus(statuses1[0], fs.makeQualified(statuses1[0].getPath()));
+ }
+
+ @Test
+ public void testListStatusOnEmptyDirectory() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path emptyDir = new Path("/emptyDir");
+ fs.mkdirs(emptyDir);
+
+ FileStatus[] statuses = fs.listStatus(emptyDir);
+ Assertions.assertThat(statuses.length).isEqualTo(0);
+ }
+
+ @Test
+ public void testContinuationTokenAcrossListStatus() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ fs.listStatus(path);
+
+ ListResponseData listResponseData = fs.getAbfsStore().getClient().listPath(
+ "/testContinuationToken", false, 1, null, getTestTracingContext(fs, true),
+ fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData.getContinuationToken()).isNotNull();
+ Assertions.assertThat(listResponseData.getFileStatusList()).hasSize(1);
+
+ ListResponseData listResponseData1 = fs.getAbfsStore().getClient().listPath(
+ "/testContinuationToken", false, 1, listResponseData.getContinuationToken(), getTestTracingContext(fs, true),
+ fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData1.getContinuationToken()).isNull();
+ Assertions.assertThat(listResponseData1.getFileStatusList()).hasSize(1);
+ }
+
+ @Test
+ public void testInvalidContinuationToken() throws Exception {
+ assumeHnsDisabled();
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testInvalidContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ intercept(AbfsRestOperationException.class,
+ () -> fs.getAbfsStore().getClient().listPath(
+ "/testInvalidContinuationToken", false, 1, "invalidToken",
+ getTestTracingContext(fs, true), fs.getAbfsStore().getUri()));
+ }
+
+ @Test
+ public void testEmptyContinuationToken() throws Exception {
+ final AzureBlobFileSystem fs = getFileSystem();
+ Path path = new Path("/testInvalidContinuationToken");
+ fs.mkdirs(path);
+ fs.create(new Path(path + "/file1"));
+ fs.create(new Path(path + "/file2"));
+
+ ListResponseData listResponseData = fs.getAbfsStore().getClient().listPath(
+ "/testInvalidContinuationToken", false, 1, "",
+ getTestTracingContext(fs, true), fs.getAbfsStore().getUri());
+
+ Assertions.assertThat(listResponseData.getContinuationToken()).isNotNull();
+ Assertions.assertThat(listResponseData.getFileStatusList()).hasSize(1);
+ }
+
+ private void assertFileFileStatus(final FileStatus fileStatus,
Review Comment:
typo in name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013904387
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1277,57 +1275,14 @@ public String listStatus(final Path path, final String startFrom,
do {
try (AbfsPerfInfo perfInfo = startTracking("listStatus", "listPath")) {
- AbfsRestOperation op = listingClient.listPath(relativePath, false,
- abfsConfiguration.getListMaxResults(), continuation,
- tracingContext);
+ ListResponseData listResponseData = listingClient.listPath(relativePath,
+ false, abfsConfiguration.getListMaxResults(), continuation,
+ tracingContext, this.uri);
+ AbfsRestOperation op = listResponseData.getOp();
perfInfo.registerResult(op.getResult());
- continuation = listingClient.getContinuationFromResponse(op.getResult());
- ListResultSchema retrievedSchema = op.getResult().getListResultSchema();
- if (retrievedSchema == null) {
- throw new AbfsRestOperationException(
- AzureServiceErrorCode.PATH_NOT_FOUND.getStatusCode(),
- AzureServiceErrorCode.PATH_NOT_FOUND.getErrorCode(),
- "listStatusAsync path not found",
- null, op.getResult());
- }
-
- long blockSize = abfsConfiguration.getAzureBlockSize();
-
- for (ListResultEntrySchema entry : retrievedSchema.paths()) {
- final String owner = identityTransformer.transformIdentityForGetRequest(entry.owner(), true, userName);
- final String group = identityTransformer.transformIdentityForGetRequest(entry.group(), false, primaryUserGroup);
- final String encryptionContext = entry.getXMsEncryptionContext();
- final FsPermission fsPermission = entry.permissions() == null
- ? new AbfsPermission(FsAction.ALL, FsAction.ALL, FsAction.ALL)
- : AbfsPermission.valueOf(entry.permissions());
- final boolean hasAcl = AbfsPermission.isExtendedAcl(entry.permissions());
-
- long lastModifiedMillis = 0;
- long contentLength = entry.contentLength() == null ? 0 : entry.contentLength();
- boolean isDirectory = entry.isDirectory() == null ? false : entry.isDirectory();
- if (entry.lastModified() != null && !entry.lastModified().isEmpty()) {
- lastModifiedMillis = DateTimeUtils.parseLastModifiedTime(
- entry.lastModified());
- }
-
- Path entryPath = new Path(File.separator + entry.name());
- entryPath = entryPath.makeQualified(this.uri, entryPath);
-
- fileStatuses.add(
- new VersionedFileStatus(
- owner,
- group,
- fsPermission,
- hasAcl,
- contentLength,
- isDirectory,
- 1,
- blockSize,
- lastModifiedMillis,
- entryPath,
- entry.eTag(),
- encryptionContext));
- }
+ continuation = listResponseData.getContinuationToken();
+ List<FileStatus> fileStatusListInCurrItr = listResponseData.getFileStatusList();
+ fileStatuses.addAll(fileStatusListInCurrItr);
Review Comment:
Good Suggestion taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2013935353
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/VersionedFileStatus.java:
##########
@@ -0,0 +1,127 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import org.apache.hadoop.fs.EtagSource;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+/**
+ * A File status with version info extracted from the etag value returned
+ * in a LIST or HEAD request.
+ * The etag is included in the java serialization.
+ */
+public class VersionedFileStatus extends FileStatus implements EtagSource {
Review Comment:
Added
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java:
##########
@@ -1780,19 +1782,19 @@ private Configuration createConfig(String producerQueueSize, String consumerMaxL
private void validateRename(AzureBlobFileSystem fs, Path src, Path dst,
boolean isSrcExist, boolean isDstExist, boolean isJsonExist)
throws IOException {
- Assertions.assertThat(fs.exists(dst))
- .describedAs("Renamed Destination directory should exist.")
- .isEqualTo(isDstExist);
Assertions.assertThat(fs.exists(new Path(src.getParent(), src.getName() + SUFFIX)))
.describedAs("Renamed Pending Json file should exist.")
.isEqualTo(isJsonExist);
Assertions.assertThat(fs.exists(src))
- .describedAs("Renamed Destination directory should exist.")
+ .describedAs("Renamed Source directory should exist.")
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014035695
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsCustomEncryption.java:
##########
@@ -301,10 +302,10 @@ private AbfsRestOperation callOperation(AzureBlobFileSystem fs,
*/
FileStatus status = fs.listStatus(testPath)[0];
Assertions.assertThat(status)
Review Comment:
Added
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsCustomEncryption.java:
##########
@@ -301,10 +302,10 @@ private AbfsRestOperation callOperation(AzureBlobFileSystem fs,
*/
FileStatus status = fs.listStatus(testPath)[0];
Assertions.assertThat(status)
- .isInstanceOf(AzureBlobFileSystemStore.VersionedFileStatus.class);
+ .isInstanceOf(VersionedFileStatus.class);
Assertions.assertThat(
Review Comment:
Added
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestListActionTaker.java:
##########
@@ -132,10 +136,10 @@ protected void addPaths(final List<Path> paths,
occurrences[0]++;
Assertions.assertThat((int) answer.getArgument(2))
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2014022080
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsCustomEncryption.java:
##########
@@ -301,10 +302,10 @@ private AbfsRestOperation callOperation(AzureBlobFileSystem fs,
*/
FileStatus status = fs.listStatus(testPath)[0];
Assertions.assertThat(status)
- .isInstanceOf(AzureBlobFileSystemStore.VersionedFileStatus.class);
+ .isInstanceOf(VersionedFileStatus.class);
Assertions.assertThat(
Review Comment:
we can add assertion statement here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#discussion_r2004822284
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:
##########
@@ -1583,26 +1590,40 @@ public Hashtable<String, String> getXMSProperties(AbfsHttpOperation result)
/**
* Parse the XML response body returned by ListBlob API on Blob Endpoint.
- * @param stream InputStream contains the response from server.
- * @return BlobListResultSchema containing the list of entries.
- * @throws IOException if parsing fails.
+ * @param result InputStream contains the response from server.
+ * @param uri to be used for path conversion.
+ * @return {@link ListResponseData}. containing listing response.
+ * @throws AzureBlobFileSystemException if parsing fails.
*/
@Override
- public ListResultSchema parseListPathResults(final InputStream stream) throws IOException {
- if (stream == null) {
- return null;
- }
+ public ListResponseData parseListPathResults(AbfsHttpOperation result, URI uri)
+ throws AzureBlobFileSystemException {
BlobListResultSchema listResultSchema;
- try {
- final SAXParser saxParser = saxParserThreadLocal.get();
- saxParser.reset();
- listResultSchema = new BlobListResultSchema();
- saxParser.parse(stream, new BlobListXmlParser(listResultSchema, getBaseUrl().toString()));
- } catch (SAXException | IOException e) {
- throw new RuntimeException(e);
+ try (InputStream stream = result.getListResultStream()) {
+ if (stream == null) {
+ return null;
+ }
+ try {
+ final SAXParser saxParser = saxParserThreadLocal.get();
+ saxParser.reset();
+ listResultSchema = new BlobListResultSchema();
+ saxParser.parse(stream,
+ new BlobListXmlParser(listResultSchema, getBaseUrl().toString()));
+ result.setListResultSchema(listResultSchema);
+ } catch (SAXException | IOException e) {
+ throw new AbfsDriverException(e);
+ }
+ } catch (IOException e) {
+ LOG.error("Unable to deserialize list results", e);
Review Comment:
Nice, taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2742841478
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 20m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 56s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 18s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/10/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 7 new + 12 unchanged - 0 fixed = 19 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 19s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 0s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 153m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense codespell detsecrets xmllint compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux ea04b4a9351f 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 21f590ab6b6d580a05e38954251123d2367e2665 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/10/testReport/ |
| Max. process+thread count | 544 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop 19234 followup [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2673456106
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 20s | | https://github.com/apache/hadoop/pull/7421 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/1/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop 19234 followup [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2687835033
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 14s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 43s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 5s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 29 new + 12 unchanged - 0 fixed = 41 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 29s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 generated 4 new + 10 unchanged - 0 fixed = 14 total (was 10) |
| -1 :x: | javadoc | 0m 25s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 generated 4 new + 10 unchanged - 0 fixed = 14 total (was 10) |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 50s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 0s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 135m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux dea5d5b7e47f 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f3c4fe678773eb785748321d3514558aaefc258a |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/2/testReport/ |
| Max. process+thread count | 530 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19474: [ABFS][FnsOverBlob] Listing Optimizations to avoid multiple iteration over list response. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7421:
URL: https://github.com/apache/hadoop/pull/7421#issuecomment-2695226615
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 10m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 11 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 57s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 38s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 35m 58s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 20s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 18 new + 12 unchanged - 0 fixed = 30 total (was 12) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| -1 :x: | javadoc | 0m 25s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10) |
| -1 :x: | javadoc | 0m 27s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10) |
| +1 :green_heart: | spotbugs | 1m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 50s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 135m 4s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7421 |
| JIRA Issue | HADOOP-19474 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b3d24f968b2c 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b37c3d2bdb5f8771b1dbad0a179be307cf3444b1 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/3/testReport/ |
| Max. process+thread count | 761 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7421/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org