You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/12/11 06:08:36 UTC
[GitHub] [hadoop] virajjasani opened a new pull request #3788: HADOOP-17725. Keep MSI tenant ID and client ID optional (ADDENDUM)
virajjasani opened a new pull request #3788:
URL: https://github.com/apache/hadoop/pull/3788
### Description of PR
Keep MSI tenant ID and client ID optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] anmolanmol1234 commented on pull request #3788: HADOOP-17725. Keep MSI tenant ID and client ID optional (ADDENDUM)
Posted by GitBox <gi...@apache.org>.
anmolanmol1234 commented on pull request #3788:
URL: https://github.com/apache/hadoop/pull/3788#issuecomment-1066418355
For MSITokenProvider, there are 4 mandatory fields that need to be provided - The clientid, authEndPoint, tenantGuid and authority, The authEndPoint and authority configs are missing here. Can you please try once by adding these config values as well ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3788: HADOOP-17725. Keep MSI tenant ID and client ID optional (ADDENDUM)
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #3788:
URL: https://github.com/apache/hadoop/pull/3788#issuecomment-991852477
Tried setting OAuth with Msi Token provider but looks like I am missing some config.
If I use these configs,
```
<property>
<name>fs.azure.account.auth.type.abfsuswest1</name>
<value>OAuth</value>
<description>The authorization type can be SharedKey, OAuth, Custom or SAS. The
default is SharedKey.</description>
</property>
<property>
<name>fs.azure.account.oauth.provider.type.abfsuswest1</name>
<value>org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider</value>
<description>The full name of token provider class name.</description>
</property>
<property>
<name>fs.azure.account.oauth2.msi.tenant.abfsuswest1</name>
<value>${value}</value>
<description>msi tenantGuid.</description>
</property>
<property>
<name>fs.azure.account.oauth2.client.id.abfsuswest1</name>
<value>${value}</value>
<description>AAD client id.</description>
</property>
```
I get this error:
```
<testcase name="test" classname="org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider" time="0">
<skipped message="Not set: fs.azure.account.oauth.provider.type"/>
</testcase>
```
However, adding `fs.azure.account.oauth.provider.type` with value `org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider` results in error:
```
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.382 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider
[ERROR] test(org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider) Time elapsed: 0.309 s <<< ERROR!
java.lang.IllegalArgumentException: Illegal character in authority at index 8: abfss://abfs-testcontainer-db25b542-c2c9-4a07-b1da-e69d40f61d36@{account name}.dfs.core.windows.net
at java.net.URI.create(URI.java:852)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:155)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3459)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:171)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3564)
at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3517)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:591)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:603)
at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.createFileSystem(AbstractAbfsIntegrationTest.java:303)
at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.setup(AbstractAbfsIntegrationTest.java:174)
```
Is this a known issue with some missing config while running test?
mvn command:
```
$ mvn -T 1C clean verify -Dtest=none -Dit.test=ITestAbfsMsiTokenProvider
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3788: HADOOP-17725. Keep MSI tenant ID and client ID optional (ADDENDUM)
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3788:
URL: https://github.com/apache/hadoop/pull/3788#issuecomment-991516098
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 6s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 56s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 58s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 93m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3788/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3788 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f282f7126c43 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 207820ed4461eaea63ba50fad0aeb84d614dc404 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3788/1/testReport/ |
| Max. process+thread count | 523 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3788/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3788: HADOOP-17725. Keep MSI tenant ID and client ID optional (ADDENDUM)
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #3788:
URL: https://github.com/apache/hadoop/pull/3788#issuecomment-991774418
Storage account: US-West
```
$ mvn test
[INFO] Results:
[INFO]
[WARNING] Tests run: 356, Failures: 0, Errors: 0, Skipped: 107
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:23 min
[INFO] Finished at: 2021-12-12T01:43:57+05:30
[INFO] ------------------------------------------------------------------------
```
HNS-Sharedkey:
```
$ ./dev-support/testrun-scripts/runtests.sh -c "HNS-SharedKey"
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.953 s
[INFO] Finished at: 2021-12-12T01:30:00+05:30
[INFO] ------------------------------------------------------------------------
Running the combination: HNS-SharedKey...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 566, Failures: 0, Errors: 0, Skipped: 566
[INFO] Results:
[INFO]
[WARNING] Tests run: 263, Failures: 0, Errors: 0, Skipped: 263
Time taken: 1 mins 4 secs.
```
However, integration tests are failing with `StorageException: The account being accessed does not support http` even though I have `enableHttpsTrafficOnly` as true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3788: HADOOP-17725. Keep MSI tenant ID and client ID optional (ADDENDUM)
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #3788:
URL: https://github.com/apache/hadoop/pull/3788#issuecomment-991774418
Storage account: US-West
```
$ mvn test
[INFO] Results:
[INFO]
[WARNING] Tests run: 356, Failures: 0, Errors: 0, Skipped: 107
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:23 min
[INFO] Finished at: 2021-12-12T01:43:57+05:30
[INFO] ------------------------------------------------------------------------
```
HNS-Sharedkey:
```
$ ./dev-support/testrun-scripts/runtests.sh -c "HNS-SharedKey"
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.953 s
[INFO] Finished at: 2021-12-12T01:30:00+05:30
[INFO] ------------------------------------------------------------------------
Running the combination: HNS-SharedKey...
----- Test results -----
[INFO] Results:
[INFO]
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO]
[WARNING] Tests run: 566, Failures: 0, Errors: 0, Skipped: 566
[INFO] Results:
[INFO]
[WARNING] Tests run: 263, Failures: 0, Errors: 0, Skipped: 263
Time taken: 1 mins 4 secs.
```
However, integration tests are failing with `StorageException: The account being accessed does not support http` even though I have `enableHttpsTrafficOnly` as true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3788: HADOOP-17725. Keep MSI tenant ID and client ID optional (ADDENDUM)
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3788:
URL: https://github.com/apache/hadoop/pull/3788#issuecomment-991516098
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 6s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 56s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 58s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not generate ASF License warnings. |
| | | 93m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3788/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3788 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f282f7126c43 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 207820ed4461eaea63ba50fad0aeb84d614dc404 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3788/1/testReport/ |
| Max. process+thread count | 523 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3788/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org