You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "manika137 (via GitHub)" <gi...@apache.org> on 2025/12/31 05:43:54 UTC
[PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
manika137 opened a new pull request, #8152:
URL: https://github.com/apache/hadoop/pull/8152
### Description of PR
We have observed cases of ABFS driver being initialised with DFS for FNS accounts which is not a recommended endpoint by the service.
To prevent this, we’re normalizing endpoint to Blob for FNS accounts even if they were initialized with DFS endpoint. We also shall restrict usage of a separate ingress service type for FNS accounts.
### How was this patch tested?
Test suite was run. Results of which will be added in the comments below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#issuecomment-3736877486
Spotbug reported is not related to the changes done in this patch.
Other PRs are also reporting same hence this will be taken up in PR https://github.com/apache/hadoop/pull/8153.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 merged PR #8152:
URL: https://github.com/apache/hadoop/pull/8152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664994832
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientHandler.java:
##########
@@ -86,11 +86,47 @@ public AbfsClientHandler(final URL baseUrl,
* Initialize the default service type based on the user configuration.
* @param abfsConfiguration set by user.
*/
- private void initServiceType(final AbfsConfiguration abfsConfiguration) {
+ public void initServiceType(final AbfsConfiguration abfsConfiguration) {
this.defaultServiceType = abfsConfiguration.getFsConfiguredServiceType();
this.ingressServiceType = abfsConfiguration.getIngressServiceType();
}
+ /**
+ * Sets the default service type.
+ *
+ * @param defaultServiceType the service type to set as default
+ */
+ public void setDefaultServiceType(AbfsServiceType defaultServiceType) {
+ this.defaultServiceType = defaultServiceType;
+ }
+
+ /**
+ * Sets the ingress service type.
+ *
+ * @param ingressServiceType the ingress service type
+ */
+ public void setIngressServiceType(AbfsServiceType ingressServiceType) {
+ this.ingressServiceType = ingressServiceType;
+ }
+
+ /**
+ * Gets the default service type.
+ *
+ * @return the default service type
+ */
+ public AbfsServiceType getDefaultServiceType() {
+ return defaultServiceType;
+ }
+
+ /**
+ * Gets the default ingress service type.
+ *
+ * @return the default ingress service type
+ */
+ public AbfsServiceType getDefaultIngressServiceType() {
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2667526088
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -798,9 +798,12 @@ private AbfsOutputStreamContext populateAbfsOutputStreamContext(
ContextEncryptionAdapter contextEncryptionAdapter,
TracingContext tracingContext) {
int bufferSize = abfsConfiguration.getWriteBufferSize();
+ getClientHandler().initServiceType(abfsConfiguration);
Review Comment:
only reqd for testing purpose. We have tests where we validate ingress handlers based on configurations. Now since we set ingress service type from clienthandler instead of abfsconfig directly- we need to reassign it.
For user flow- it still remains the same since config wont change in the middle of workload
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#issuecomment-3719323721
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 21m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 20s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/2/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 22s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 0m 31s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 2 new + 1585 unchanged - 0 fixed = 1587 total (was 1585) |
| -1 :x: | javadoc | 0m 30s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 2 new + 1464 unchanged - 0 fixed = 1466 total (was 1464) |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 13s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 6s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 125m 7s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8152 |
| JIRA Issue | HADOOP-19766 |
| Optional Tests | dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs compile javac javadoc mvninstall shadedclient spotbugs checkstyle markdownlint |
| uname | Linux 73b1e32257dd 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / be844c8d3fc8f3f8f2d0438a2145f1ffa008d665 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/2/testReport/ |
| Max. process+thread count | 634 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/2/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2665013847
##########
hadoop-tools/hadoop-azure/src/site/markdown/index.md:
##########
@@ -838,6 +837,58 @@ requests. User can specify them as fixed SAS Token to be used across all the req
</property>
```
+ - ABFS allows you to implement your custom SAS Token Provider. The declared class must implement
Review Comment:
Better readability since its less scattered now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664977154
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##########
@@ -752,6 +752,15 @@ public AbfsServiceType getFsConfiguredServiceType() {
return getCaseInsensitiveEnum(FS_AZURE_FNS_ACCOUNT_SERVICE_TYPE, fsConfiguredServiceType);
}
+ /**
+ * Returns the service type identified from the URL used to initialize the FileSystem.
+ *
+ * @return the configured AbfsServiceType from the URL
+ */
+ public AbfsServiceType getFsConfiguredServiceTypeFromURL() {
+ return fsConfiguredServiceType;
Review Comment:
taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664622142
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java:
##########
@@ -850,6 +831,39 @@ public void testAuthTypeProviderSetup(AuthType authType) throws Exception {
fs.close();
}
+ /**
+ * Test to verify that when initializing a filesystem with a DFS endpoint for a FNS account,
+ * we force to Blob endpoint internally.
+ *
+ * @throws Exception if the test fails
+ */
+ @Test
+ public void testFNSDfsUsesBlobInstance() throws Exception {
Review Comment:
Ingress type validations should also be added that no matter what user configures for FNS it should be BLOB and also around the changes made in AbfsOutputStream class for dfstoBlobFallback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2680855709
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ListResponseData.java:
##########
@@ -35,6 +35,13 @@ public class ListResponseData {
private AbfsRestOperation executedRestOperation;
private String continuationToken;
+ /**
Review Comment:
Is this a requirement from Javadoc report?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#issuecomment-3701628025
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 27s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 34m 58s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/1/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 23s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/1/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/1/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 22s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 5 new + 4 unchanged - 0 fixed = 9 total (was 4) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 0m 30s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/1/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 1585 unchanged - 0 fixed = 1586 total (was 1585) |
| -1 :x: | javadoc | 0m 30s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/1/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 1 new + 1464 unchanged - 0 fixed = 1465 total (was 1464) |
| +1 :green_heart: | spotbugs | 1m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 54s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 6s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 105m 13s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8152 |
| JIRA Issue | HADOOP-19766 |
| Optional Tests | dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs compile javac javadoc mvninstall shadedclient spotbugs checkstyle markdownlint |
| uname | Linux 77484494ca13 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c40d9e040bc1bcae87e321682d5dbe55d36266ef |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/1/testReport/ |
| Max. process+thread count | 637 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/1/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664552789
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1906,6 +1909,17 @@ private AbfsPerfInfo startTracking(String callerName, String calleeName) {
return new AbfsPerfInfo(abfsPerfTracker, callerName, calleeName);
}
+ /**
+ * Resets all service types to use BLOB.
+ * Updates the client to reflect the new default service type.
+ */
+ public void resetEndpointforFNS() {
+ getAbfsConfiguration().setFsConfiguredServiceTypetoBlob();
+ getClientHandler().setDefaultServiceType(AbfsServiceType.BLOB);
Review Comment:
Repeated use of gets, can be initialized in a variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664589265
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java:
##########
@@ -42,15 +45,16 @@
import static java.net.HttpURLConnection.HTTP_UNAVAILABLE;
import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_IS_HNS_ENABLED;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_INGRESS_SERVICE_TYPE;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.accountProperty;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.*;
Review Comment:
* imports should be reverted back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664384142
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -315,6 +316,14 @@ public void initialize(URI uri, Configuration configuration)
throw new InvalidConfigurationValueException(FS_AZURE_ACCOUNT_IS_HNS_ENABLED, ex);
}
+ // For FNS-DFS accounts, reset the endpoint to Blob and update the tracing
Review Comment:
Nit: use block comment as above
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java:
##########
@@ -160,4 +164,53 @@ public void testFileSystemInitFailsIfNotAbleToDetermineAccountType() throws Exce
FS_AZURE_ACCOUNT_IS_HNS_ENABLED, () ->
mockedFs.initialize(fs.getUri(), getRawConfiguration()));
}
+
+ /**
+ * Test to verify that the fnsEndptConvertedIndicator ("T") is present in the tracing header
+ * after endpoint conversion during AzureBlobFileSystem initialization.
+ *
+ * @throws Exception if any error occurs during the test
+ */
+ @Test
+ public void testFNSEndptConvertedIndicatorInHeader() throws Exception {
+ assumeHnsDisabled();
+ String scheme = "abfs";
+ String dfsDomain = "dfs.core.windows.net";
+ String endptConversionIndicatorInTc = "T";
+ Configuration conf = new Configuration(getRawConfiguration());
+ conf.setBoolean(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION, true);
+
+ String dfsUri = String.format("%s://%s@%s.%s/",
+ scheme, getFileSystemName(),
+ getAccountName().substring(0, getAccountName().indexOf(DOT)),
+ dfsDomain);
+
+ // Initialize filesystem with DFS endpoint
+ AzureBlobFileSystem fs =
Review Comment:
Should be inside try to autoclose
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##########
@@ -752,6 +752,15 @@ public AbfsServiceType getFsConfiguredServiceType() {
return getCaseInsensitiveEnum(FS_AZURE_FNS_ACCOUNT_SERVICE_TYPE, fsConfiguredServiceType);
}
+ /**
+ * Returns the service type identified from the URL used to initialize the FileSystem.
+ *
+ * @return the configured AbfsServiceType from the URL
+ */
+ public AbfsServiceType getFsConfiguredServiceTypeFromURL() {
+ return fsConfiguredServiceType;
Review Comment:
Nit: Better to change this variable name also to `fsConfiguredServiceTypeFromUrl`
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##########
@@ -1799,6 +1808,14 @@ void setReadAheadEnabled(final boolean enabledReadAhead) {
this.enabledReadAhead = enabledReadAhead;
}
+ /**
+ * Sets the configured service type to BLOB.
+ * Majorly required to correctly set user agent for FNS-Blob
+ */
+ void setFsConfiguredServiceTypetoBlob() {
Review Comment:
Nit: camel case in method name
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1906,6 +1909,17 @@ private AbfsPerfInfo startTracking(String callerName, String calleeName) {
return new AbfsPerfInfo(abfsPerfTracker, callerName, calleeName);
}
+ /**
+ * Resets all service types to use BLOB.
+ * Updates the client to reflect the new default service type.
+ */
+ public void resetEndpointforFNS() {
Review Comment:
Nit: camel case in method name
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##########
@@ -1799,6 +1808,14 @@ void setReadAheadEnabled(final boolean enabledReadAhead) {
this.enabledReadAhead = enabledReadAhead;
}
+ /**
+ * Sets the configured service type to BLOB.
+ * Majorly required to correctly set user agent for FNS-Blob
+ */
+ void setFsConfiguredServiceTypetoBlob() {
Review Comment:
Better to have generic method that can accept Service Type and caller can call it with Blob
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -156,6 +156,7 @@ public class AzureBlobFileSystem extends FileSystem
*/
private boolean isClosed = true;
private final String fileSystemId = UUID.randomUUID().toString();
+ private final String DFS_DOMAIN_INDICATOR = ".dfs.";
Review Comment:
+1
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientHandler.java:
##########
@@ -86,11 +86,47 @@ public AbfsClientHandler(final URL baseUrl,
* Initialize the default service type based on the user configuration.
* @param abfsConfiguration set by user.
*/
- private void initServiceType(final AbfsConfiguration abfsConfiguration) {
+ public void initServiceType(final AbfsConfiguration abfsConfiguration) {
this.defaultServiceType = abfsConfiguration.getFsConfiguredServiceType();
this.ingressServiceType = abfsConfiguration.getIngressServiceType();
}
+ /**
+ * Sets the default service type.
+ *
+ * @param defaultServiceType the service type to set as default
+ */
+ public void setDefaultServiceType(AbfsServiceType defaultServiceType) {
+ this.defaultServiceType = defaultServiceType;
+ }
+
+ /**
+ * Sets the ingress service type.
+ *
+ * @param ingressServiceType the ingress service type
+ */
+ public void setIngressServiceType(AbfsServiceType ingressServiceType) {
+ this.ingressServiceType = ingressServiceType;
+ }
+
+ /**
+ * Gets the default service type.
+ *
+ * @return the default service type
+ */
+ public AbfsServiceType getDefaultServiceType() {
+ return defaultServiceType;
+ }
+
+ /**
+ * Gets the default ingress service type.
+ *
+ * @return the default ingress service type
+ */
+ public AbfsServiceType getDefaultIngressServiceType() {
Review Comment:
methond name should be `getngressServiceType()`
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientHandler.java:
##########
@@ -86,11 +86,47 @@ public AbfsClientHandler(final URL baseUrl,
* Initialize the default service type based on the user configuration.
* @param abfsConfiguration set by user.
*/
- private void initServiceType(final AbfsConfiguration abfsConfiguration) {
+ public void initServiceType(final AbfsConfiguration abfsConfiguration) {
this.defaultServiceType = abfsConfiguration.getFsConfiguredServiceType();
this.ingressServiceType = abfsConfiguration.getIngressServiceType();
}
+ /**
+ * Sets the default service type.
+ *
+ * @param defaultServiceType the service type to set as default
+ */
+ public void setDefaultServiceType(AbfsServiceType defaultServiceType) {
+ this.defaultServiceType = defaultServiceType;
+ }
+
+ /**
+ * Sets the ingress service type.
+ *
+ * @param ingressServiceType the ingress service type
+ */
+ public void setIngressServiceType(AbfsServiceType ingressServiceType) {
+ this.ingressServiceType = ingressServiceType;
+ }
+
+ /**
+ * Gets the default service type.
+ *
+ * @return the default service type
+ */
+ public AbfsServiceType getDefaultServiceType() {
Review Comment:
Not used, can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2670943006
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -798,9 +798,12 @@ private AbfsOutputStreamContext populateAbfsOutputStreamContext(
ContextEncryptionAdapter contextEncryptionAdapter,
TracingContext tracingContext) {
int bufferSize = abfsConfiguration.getWriteBufferSize();
+ getClientHandler().initServiceType(abfsConfiguration);
Review Comment:
This was purely was test purpose. We had some tests that validate ingress handlers, clients by changing the service type using the configuration
Now since we use clientHandler as source of truth for these values- we were trying to reset it here. For users- nothing changes since they cant change the config value mid-workload
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2670945610
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -798,9 +798,12 @@ private AbfsOutputStreamContext populateAbfsOutputStreamContext(
ContextEncryptionAdapter contextEncryptionAdapter,
TracingContext tracingContext) {
int bufferSize = abfsConfiguration.getWriteBufferSize();
+ getClientHandler().initServiceType(abfsConfiguration);
Review Comment:
Rethought on this- better to remove it and change the tests to set service types through clienthandler instead before validating
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#issuecomment-3728223391
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 21m 44s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 54s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 19s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/4/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 27m 25s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| -1 :x: | javadoc | 0m 30s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/4/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 1584 unchanged - 1 fixed = 1585 total (was 1585) |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | spotbugs | 1m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 4s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not generate ASF License warnings. |
| | | 125m 40s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8152 |
| JIRA Issue | HADOOP-19766 |
| Optional Tests | dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs compile javac javadoc mvninstall shadedclient spotbugs checkstyle markdownlint |
| uname | Linux 75bc8e8de4b7 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 27d2e88f82bdb387457e0b1da6ed471223f1a5d4 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/4/testReport/ |
| Max. process+thread count | 786 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/4/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2667306940
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java:
##########
@@ -160,4 +164,53 @@ public void testFileSystemInitFailsIfNotAbleToDetermineAccountType() throws Exce
FS_AZURE_ACCOUNT_IS_HNS_ENABLED, () ->
mockedFs.initialize(fs.getUri(), getRawConfiguration()));
}
+
+ /**
+ * Test to verify that the fnsEndptConvertedIndicator ("T") is present in the tracing header
+ * after endpoint conversion during AzureBlobFileSystem initialization.
+ *
+ * @throws Exception if any error occurs during the test
+ */
+ @Test
+ public void testFNSEndptConvertedIndicatorInHeader() throws Exception {
+ assumeHnsDisabled();
+ String scheme = "abfs";
+ String dfsDomain = "dfs.core.windows.net";
+ String endptConversionIndicatorInTc = "T";
+ Configuration conf = new Configuration(getRawConfiguration());
+ conf.setBoolean(AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION, true);
+
+ String dfsUri = String.format("%s://%s@%s.%s/",
+ scheme, getFileSystemName(),
+ getAccountName().substring(0, getAccountName().indexOf(DOT)),
+ dfsDomain);
+
+ // Initialize filesystem with DFS endpoint
+ AzureBlobFileSystem fs =
Review Comment:
taken
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java:
##########
@@ -850,6 +831,39 @@ public void testAuthTypeProviderSetup(AuthType authType) throws Exception {
fs.close();
}
+ /**
+ * Test to verify that when initializing a filesystem with a DFS endpoint for a FNS account,
+ * we force to Blob endpoint internally.
+ *
+ * @throws Exception if the test fails
+ */
+ @Test
+ public void testFNSDfsUsesBlobInstance() throws Exception {
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2665012927
##########
hadoop-tools/hadoop-azure/src/site/markdown/index.md:
##########
@@ -838,6 +837,58 @@ requests. User can specify them as fixed SAS Token to be used across all the req
</property>
```
+ - ABFS allows you to implement your custom SAS Token Provider. The declared class must implement
Review Comment:
No. We had earlier mentioned to add configs for both OAuth and delegated SAS by referring to the sections above. For SAS however, the configs remain constant since we dont have multiple implementations of it (unlike OAuth).
Would be better to mention all fixed configs ones at one place so user only needs to check configs for OAuth implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2667953490
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -156,6 +156,7 @@ public class AzureBlobFileSystem extends FileSystem
*/
private boolean isClosed = true;
private final String fileSystemId = UUID.randomUUID().toString();
+ private final String DFS_DOMAIN_INDICATOR = ".dfs.";
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664981186
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java:
##########
@@ -1799,6 +1808,14 @@ void setReadAheadEnabled(final boolean enabledReadAhead) {
this.enabledReadAhead = enabledReadAhead;
}
+ /**
+ * Sets the configured service type to BLOB.
+ * Majorly required to correctly set user agent for FNS-Blob
+ */
+ void setFsConfiguredServiceTypetoBlob() {
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2667526088
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -798,9 +798,12 @@ private AbfsOutputStreamContext populateAbfsOutputStreamContext(
ContextEncryptionAdapter contextEncryptionAdapter,
TracingContext tracingContext) {
int bufferSize = abfsConfiguration.getWriteBufferSize();
+ getClientHandler().initServiceType(abfsConfiguration);
Review Comment:
only reqd for testing purpose. We have tests where we validate ingress handlers based on configurations. Now since we set ingress service type from clienthandler instead of abfsconfig directly- we need to reassign it.
For user flow- it still remains the same since config wont change in the middle of workload
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664562808
##########
hadoop-tools/hadoop-azure/src/site/markdown/index.md:
##########
@@ -838,6 +837,58 @@ requests. User can specify them as fixed SAS Token to be used across all the req
</property>
```
+ - ABFS allows you to implement your custom SAS Token Provider. The declared class must implement
Review Comment:
Was this expected to be covered as part of this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2667551570
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java:
##########
@@ -42,15 +45,16 @@
import static java.net.HttpURLConnection.HTTP_UNAVAILABLE;
import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_IS_HNS_ENABLED;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_INGRESS_SERVICE_TYPE;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.accountProperty;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.*;
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2680867234
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ListResponseData.java:
##########
@@ -35,6 +35,13 @@ public class ListResponseData {
private AbfsRestOperation executedRestOperation;
private String continuationToken;
+ /**
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#issuecomment-3704510777
# Test Results
============================================================
HNS-OAuth-DFS
============================================================
[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus.testListPath -- Time elapsed: 65.43 s <<< FAILURE!
[WARNING] Tests run: 223, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 892, Failures: 0, Errors: 0, Skipped: 222
[ERROR] Tests run: 158, Failures: 1, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23
============================================================
HNS-SharedKey-DFS
============================================================
[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus.testListPath -- Time elapsed: 54.55 s <<< FAILURE!
[WARNING] Tests run: 223, Failures: 0, Errors: 0, Skipped: 5
[WARNING] Tests run: 895, Failures: 0, Errors: 0, Skipped: 168
[ERROR] Tests run: 158, Failures: 1, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 10
============================================================
AppendBlob-HNS-OAuth-DFS
============================================================
[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus.testListPath -- Time elapsed: 49.64 s <<< FAILURE!
[WARNING] Tests run: 223, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 892, Failures: 0, Errors: 0, Skipped: 233
[ERROR] Tests run: 135, Failures: 1, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23
============================================================
NonHNS-SharedKey-Blob
============================================================
[WARNING] Tests run: 223, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 742, Failures: 0, Errors: 0, Skipped: 157
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11
============================================================
NonHNS-OAuth-Blob
============================================================
[WARNING] Tests run: 223, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 739, Failures: 0, Errors: 0, Skipped: 158
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
============================================================
AppendBlob-NonHNS-OAuth-Blob
============================================================
[WARNING] Tests run: 223, Failures: 0, Errors: 0, Skipped: 11
[WARNING] Tests run: 734, Failures: 0, Errors: 0, Skipped: 204
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
============================================================
HNS-Oauth-DFS-IngressBlob
============================================================
[ERROR] org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemListStatus.testListPath -- Time elapsed: 60.37 s <<< FAILURE!
[WARNING] Tests run: 223, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 766, Failures: 0, Errors: 0, Skipped: 231
[ERROR] Tests run: 158, Failures: 1, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23
The test testListPath is fixed in PR https://github.com/apache/hadoop/pull/8137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2660313963
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -156,6 +156,7 @@ public class AzureBlobFileSystem extends FileSystem
*/
private boolean isClosed = true;
private final String fileSystemId = UUID.randomUUID().toString();
+ private final String DFS_DOMAIN_INDICATOR = ".dfs.";
Review Comment:
We have a constant already defined for this, we can use that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2667251994
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java:
##########
@@ -315,6 +316,14 @@ public void initialize(URI uri, Configuration configuration)
throw new InvalidConfigurationValueException(FS_AZURE_ACCOUNT_IS_HNS_ENABLED, ex);
}
+ // For FNS-DFS accounts, reset the endpoint to Blob and update the tracing
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664639075
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -798,9 +798,12 @@ private AbfsOutputStreamContext populateAbfsOutputStreamContext(
ContextEncryptionAdapter contextEncryptionAdapter,
TracingContext tracingContext) {
int bufferSize = abfsConfiguration.getWriteBufferSize();
+ getClientHandler().initServiceType(abfsConfiguration);
Review Comment:
Why is it needed? During client handler init we do initialize the service types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664590422
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java:
##########
@@ -42,15 +45,16 @@
import static java.net.HttpURLConnection.HTTP_UNAVAILABLE;
import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_IS_HNS_ENABLED;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.FS_AZURE_INGRESS_SERVICE_TYPE;
-import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.accountProperty;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.*;
+import static org.apache.hadoop.fs.azurebfs.constants.ConfigurationKeys.*;
import static org.apache.hadoop.fs.azurebfs.constants.FileSystemUriSchemes.ABFS_BLOB_DOMAIN_NAME;
import static org.apache.hadoop.fs.azurebfs.constants.FileSystemUriSchemes.ABFS_DFS_DOMAIN_NAME;
import static org.apache.hadoop.fs.azurebfs.services.AbfsErrors.INCORRECT_INGRESS_TYPE;
+import static org.apache.hadoop.fs.azurebfs.services.RetryPolicyConstants.EXPONENTIAL_RETRY_POLICY_ABBREVIATION;
import static org.apache.hadoop.test.LambdaTestUtils.intercept;
import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.*;
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2664993589
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientHandler.java:
##########
@@ -86,11 +86,47 @@ public AbfsClientHandler(final URL baseUrl,
* Initialize the default service type based on the user configuration.
* @param abfsConfiguration set by user.
*/
- private void initServiceType(final AbfsConfiguration abfsConfiguration) {
+ public void initServiceType(final AbfsConfiguration abfsConfiguration) {
this.defaultServiceType = abfsConfiguration.getFsConfiguredServiceType();
this.ingressServiceType = abfsConfiguration.getIngressServiceType();
}
+ /**
+ * Sets the default service type.
+ *
+ * @param defaultServiceType the service type to set as default
+ */
+ public void setDefaultServiceType(AbfsServiceType defaultServiceType) {
+ this.defaultServiceType = defaultServiceType;
+ }
+
+ /**
+ * Sets the ingress service type.
+ *
+ * @param ingressServiceType the ingress service type
+ */
+ public void setIngressServiceType(AbfsServiceType ingressServiceType) {
+ this.ingressServiceType = ingressServiceType;
+ }
+
+ /**
+ * Gets the default service type.
+ *
+ * @return the default service type
+ */
+ public AbfsServiceType getDefaultServiceType() {
Review Comment:
Taken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#issuecomment-3721849090
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 19s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/3/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 18s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| -1 :x: | javadoc | 0m 31s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 1585 unchanged - 0 fixed = 1586 total (was 1585) |
| -1 :x: | javadoc | 0m 30s | [/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 1 new + 1464 unchanged - 0 fixed = 1465 total (was 1464) |
| +1 :green_heart: | spotbugs | 1m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 5s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 6s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 105m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8152 |
| JIRA Issue | HADOOP-19766 |
| Optional Tests | dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs compile javac javadoc mvninstall shadedclient spotbugs checkstyle markdownlint |
| uname | Linux 09d68f52ea99 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7cc680a3626fd99e16ca755ccb90ec48603d3b88 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/3/testReport/ |
| Max. process+thread count | 634 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/3/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2675116652
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1906,6 +1908,24 @@ private AbfsPerfInfo startTracking(String callerName, String calleeName) {
return new AbfsPerfInfo(abfsPerfTracker, callerName, calleeName);
}
+ /**
+ * Restricts the service type to BLOB for the current client handler and configuration when FNS account detected
+ * This method sets the ingress service type and the configured service type to {@code AbfsServiceType.BLOB}.
+ */
+ public void restrictServiceTypeToBlob() {
Review Comment:
Why 2 different methods?
Can't we combine them. Any way they are called together
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2675804792
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:
##########
@@ -1906,6 +1908,24 @@ private AbfsPerfInfo startTracking(String callerName, String calleeName) {
return new AbfsPerfInfo(abfsPerfTracker, callerName, calleeName);
}
+ /**
+ * Restricts the service type to BLOB for the current client handler and configuration when FNS account detected
+ * This method sets the ingress service type and the configured service type to {@code AbfsServiceType.BLOB}.
+ */
+ public void restrictServiceTypeToBlob() {
Review Comment:
Makes sense. Combined them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#issuecomment-3728870067
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 34s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 36s | | trunk passed |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 19s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/5/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 14s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 31s | | hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 0 new + 1584 unchanged - 1 fixed = 1584 total (was 1585) |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | spotbugs | 1m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 52s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 105m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8152 |
| JIRA Issue | HADOOP-19766 |
| Optional Tests | dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs compile javac javadoc mvninstall shadedclient spotbugs checkstyle markdownlint |
| uname | Linux 1e39531f4017 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 464997a60fed2e57f58ae46c9315367a8255b75c |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/5/testReport/ |
| Max. process+thread count | 640 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8152/5/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19766: ABFS. Force Blob Instance for FNS Accounts [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8152:
URL: https://github.com/apache/hadoop/pull/8152#discussion_r2675574018
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingContext.java:
##########
@@ -67,6 +67,8 @@ public class TracingContext {
//final concatenated ID list set into x-ms-client-request-id header
private String header = EMPTY_STRING;
private String ingressHandler = EMPTY_STRING;
+ private Boolean fnsEndpointConverted = false;
+ private String fnsEndptConvertedIndicator = "T";
Review Comment:
Add comment to specify reason to choose T as an indicator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org