You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "adideshpande (via GitHub)" <gi...@apache.org> on 2025/02/11 18:23:46 UTC
[PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
adideshpande opened a new pull request, #7379:
URL: https://github.com/apache/hadoop/pull/7379
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
- Continues work on https://github.com/apache/hadoop/pull/6550
- Addressed the review comments and added updates to the Javadoc and documentation for s3a custom headers.
Description of original PR:
```
Support to add custom headers to s3a s3 and sts client
Introducing new custom.headers config for each client type, Upon setting this header each s3/sts request will set these custom headers by setting overrideClientConfig on clients
"fs.s3a.client.sts.custom.headers" = "Header1:Value1:Value2,Header2:Value1"
"fs.s3a.client.s3.custom.headers" = "Header1:Value1:Value2,Header2:Value1"
```
### How was this patch tested?
Ran integ tests using `mvn -Dparallel-tests -DtestsThreadCount=8 clean verify` for a bucket in us-west-2
UTs added which succeeded as well
### For code changes:
- [ Y ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ Y ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ N/A ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ N/A ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2986639628
I filed #7750 with a fix for the compilation error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2651887469
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 1s | | trunk passed |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 50s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 2s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| -1 :x: | javadoc | 0m 17s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 18s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| +1 :green_heart: | spotbugs | 0m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 43s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 36s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not generate ASF License warnings. |
| | | 72m 51s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7379 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux f8c659c85ddf 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3858e334c759a66f37f48794626a70d5b90ac533 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/1/testReport/ |
| Max. process+thread count | 552 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "adideshpande (via GitHub)" <gi...@apache.org>.
adideshpande commented on code in PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#discussion_r2082319297
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md:
##########
@@ -947,6 +947,32 @@ The switch to turn S3A auditing on or off.
</property>
```
+
+### Configuring Custom Headers for AWS Service Clients
+
+The S3A client allows users to set custom headers for specific AWS services, such as S3 and STS.
Review Comment:
Sure thanks for the suggestion, rephrased it in the new revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2988756402
@cnauroth missed that, did it myself as soon as I noticed the failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "adideshpande (via GitHub)" <gi...@apache.org>.
adideshpande commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2981996737
Hi @steveloughran gentle ping for reviewing this. @ahmarsuhail has already approved the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "adideshpande (via GitHub)" <gi...@apache.org>.
adideshpande commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2845236949
@ahmarsuhail Sure, I have rebased it on top of origin/trunk and it looks cleaner now with only 4 file changes (and 9 commits). Could you please take a look? Let me know if you need this to be rebased into a single commit. Thanks.
P.S: My local repo had become messy while I was trying to fetch and merge a few commits, and fix some conflicts. I had to nuke it and checkout the repo locally again :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "adideshpande (via GitHub)" <gi...@apache.org>.
adideshpande commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2689249444
@ahmarsuhail requesting your review on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "ahmarsuhail (via GitHub)" <gi...@apache.org>.
ahmarsuhail commented on code in PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#discussion_r2081836752
##########
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md:
##########
@@ -947,6 +947,32 @@ The switch to turn S3A auditing on or off.
</property>
```
+
+### Configuring Custom Headers for AWS Service Clients
+
+The S3A client allows users to set custom headers for specific AWS services, such as S3 and STS.
Review Comment:
nit: you can just rephrase this to:
"You can set custom headers for S3 and STS requests. These headers are set on client level, and will be sent for all requests made to these services."
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##########
@@ -1339,6 +1340,37 @@ private Constants() {
public static final String AWS_SERVICE_IDENTIFIER_DDB = "DDB";
public static final String AWS_SERVICE_IDENTIFIER_STS = "STS";
+ /** Prefix for S3A client-specific properties.
+ * value: {@value}
+ */
+ public static final String FS_S3A_CLIENT_PREFIX = "fs.s3a.client.";
+
+ /** Custom headers postfix.
+ * value: {@value}
+ */
+ public static final String CUSTOM_HEADERS_POSTFIX = ".custom.headers";
+
+ /**
Review Comment:
look at steve's comment on the other PR and fix: https://github.com/apache/hadoop/pull/6550/files#r1812732876
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2845367568
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 20s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 54s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 13s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 15s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 0m 15s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 0m 12s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| -1 :x: | javac | 0m 12s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| -1 :x: | mvnsite | 0m 16s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | javadoc | 0m 13s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 17s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| -1 :x: | spotbugs | 0m 14s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 21m 19s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 17s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not generate ASF License warnings. |
| | | 70m 11s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7379 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 0aa1732dcfeb 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2c7b2375b4b0f819cd05e1d18bcba46698f66a1d |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/testReport/ |
| Max. process+thread count | 556 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#discussion_r2082410937
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java:
##########
@@ -412,6 +419,36 @@ private static void initSigner(Configuration conf,
}
}
+ /**
+ * Initialize custom request headers for AWS clients.
+ * @param conf hadoop configuration
+ * @param clientConfig client configuration to update
+ * @param awsServiceIdentifier service name
+ */
+ private static void initRequestHeaders(Configuration conf,
+ ClientOverrideConfiguration.Builder clientConfig, String awsServiceIdentifier) {
+ String configKey = null;
+ switch (awsServiceIdentifier) {
+ case AWS_SERVICE_IDENTIFIER_S3:
+ configKey = CUSTOM_HEADERS_S3;
+ break;
+ case AWS_SERVICE_IDENTIFIER_STS:
+ configKey = CUSTOM_HEADERS_STS;
+ break;
+ default:
+ // No known service.
+ }
+ if (configKey != null) {
+ Map<String, String> awsClientCustomHeadersMap =
+ S3AUtils.getTrimmedStringCollectionSplitByEquals(conf, configKey);
+ awsClientCustomHeadersMap.forEach((header, valueString) -> {
+ List<String> headerValues = Arrays.asList(valueString.split(";"));
Review Comment:
trim the header values of whitespace? or is there some risk that this could ever be needed?
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java:
##########
@@ -201,4 +208,64 @@ public void testCreateApiConnectionSettingsDefault() {
private void setOptionsToValue(String value, Configuration conf, String... keys) {
Arrays.stream(keys).forEach(key -> conf.set(key, value));
}
+
+ /**
+ * if {@link org.apache.hadoop.fs.s3a.Constants#CUSTOM_HEADERS_STS} is set,
+ * verify that returned client configuration has desired headers set.
+ */
+ @Test
+ public void testInitRequestHeadersForSTS() throws IOException {
+ final Configuration conf = new Configuration();
+ conf.set(CUSTOM_HEADERS_STS, "foo=bar;baz,qux=quux");
+ Assertions.assertThat(conf.get(CUSTOM_HEADERS_S3))
+ .describedAs("Custom client headers for s3 %s", CUSTOM_HEADERS_S3)
+ .isNull();
+
+ Assertions.assertThat(createClientConfigBuilder(conf, AWS_SERVICE_IDENTIFIER_S3)
+ .headers().size())
+ .describedAs("Count of S3 client headers")
+ .isEqualTo(0);
+ Assertions.assertThat(createClientConfigBuilder(conf, AWS_SERVICE_IDENTIFIER_STS)
+ .headers().size())
+ .describedAs("Count of STS client headers")
+ .isEqualTo(2);
+ Assertions.assertThat(createClientConfigBuilder(conf, AWS_SERVICE_IDENTIFIER_STS)
+ .headers().get("foo"))
+ .describedAs("STS client 'foo' header value")
+ .isEqualTo(Lists.newArrayList("bar", "baz"));
+ Assertions.assertThat(createClientConfigBuilder(conf, AWS_SERVICE_IDENTIFIER_STS)
+ .headers().get("qux"))
+ .describedAs("STS client 'qux' header value")
+ .isEqualTo(Lists.newArrayList("quux"));
+ }
+
+ /**
+ * if {@link org.apache.hadoop.fs.s3a.Constants#CUSTOM_HEADERS_S3} is set,
+ * verify that returned client configuration has desired headers set.
+ */
+ @Test
+ public void testInitRequestHeadersForS3() throws IOException {
+ final Configuration conf = new Configuration();
+ conf.set(CUSTOM_HEADERS_S3, "foo=bar;baz,qux=quux");
Review Comment:
repeat with a separate test adding spaces between all symbols, with assertions on the values. if we trim. it should be similar to this
Add separate tests for
```
foo=,qux=quux
foo=;,qux=quux;
foo=;
foo=,
foo=;bar
=
foo=;;
foo=duplicate;duplicate // expect only one value
foo=1,foo=2 // which one is picked up
```
there's enough here to consider adding some common assert, e.g.
```
assertHeaderValues(string, List<string> header1_values, List<string> header_2_values) {
// assert on each set of values
}
```
Also, let's cut foo/bar stuff.
How about:
"header1, header2", "value1, value2, value3"
This will be good when debugging assertion failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2688985444
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 21s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 52s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 4s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 40s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 37s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not generate ASF License warnings. |
| | | 73m 48s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7379 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux c191377c67cd 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 411eae228a662335ed14aed9cb2f0fb21d22c161 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/3/testReport/ |
| Max. process+thread count | 546 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "adideshpande (via GitHub)" <gi...@apache.org>.
adideshpande commented on code in PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#discussion_r2116866072
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/AWSClientConfig.java:
##########
@@ -412,6 +419,36 @@ private static void initSigner(Configuration conf,
}
}
+ /**
+ * Initialize custom request headers for AWS clients.
+ * @param conf hadoop configuration
+ * @param clientConfig client configuration to update
+ * @param awsServiceIdentifier service name
+ */
+ private static void initRequestHeaders(Configuration conf,
+ ClientOverrideConfiguration.Builder clientConfig, String awsServiceIdentifier) {
+ String configKey = null;
+ switch (awsServiceIdentifier) {
+ case AWS_SERVICE_IDENTIFIER_S3:
+ configKey = CUSTOM_HEADERS_S3;
+ break;
+ case AWS_SERVICE_IDENTIFIER_STS:
+ configKey = CUSTOM_HEADERS_STS;
+ break;
+ default:
+ // No known service.
+ }
+ if (configKey != null) {
+ Map<String, String> awsClientCustomHeadersMap =
+ S3AUtils.getTrimmedStringCollectionSplitByEquals(conf, configKey);
+ awsClientCustomHeadersMap.forEach((header, valueString) -> {
+ List<String> headerValues = Arrays.asList(valueString.split(";"));
Review Comment:
I don't see a risk. Trimmed header values of whitespace in the latest rev.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "govindm (via GitHub)" <gi...@apache.org>.
govindm commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2982012063
@steveloughran Could you please review this PR so it can be merged? If you're unavailable, could someone else take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2895384552
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 30m 23s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/branch-mvninstall-root.txt) | root in trunk failed. |
| -1 :x: | compile | 0m 24s | [/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt) | hadoop-aws in trunk failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 26s | [/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/branch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt) | hadoop-aws in trunk failed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09. |
| -0 :warning: | checkstyle | 0m 24s | [/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/buildtool-branch-checkstyle-hadoop-tools_hadoop-aws.txt) | The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 26s | [/branch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in trunk failed. |
| -1 :x: | javadoc | 0m 25s | [/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt) | hadoop-aws in trunk failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 25s | [/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt) | hadoop-aws in trunk failed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09. |
| -1 :x: | spotbugs | 0m 25s | [/branch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/branch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in trunk failed. |
| +1 :green_heart: | shadedclient | 3m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 20s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javac | 0m 20s | [/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | compile | 0m 21s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09. |
| -1 :x: | javac | 0m 21s | [/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 19s | [/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt) | The patch fails to run checkstyle in hadoop-aws |
| -1 :x: | mvnsite | 0m 21s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | javadoc | 0m 20s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 21s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09. |
| -1 :x: | spotbugs | 0m 20s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 3m 59s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 20s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +0 :ok: | asflicense | 0m 21s | | ASF License check generated no output? |
| | | 41m 55s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7379 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux e7ea008a14f8 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8eba46b8b3e2c5faec09d48fa78c8dd34a542318 |
| Default Java | Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/testReport/ |
| Max. process+thread count | 65 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/8/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2846095520
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 23s | | trunk passed |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 15s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 49s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not generate ASF License warnings. |
| | | 74m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7379 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 5d4f3dbc4d5d 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7ebefdec9549aa0227b3f837ad9ebd723da0d522 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/6/testReport/ |
| Max. process+thread count | 545 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2923716798
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 9m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 51s | | trunk passed |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | trunk passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 15s | | the patch passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 10s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/9/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 2 new + 3 unchanged - 0 fixed = 5 total (was 3) |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 0m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 11s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 50s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not generate ASF License warnings. |
| | | 83m 57s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.50 ServerAPI=1.50 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7379 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 47a7067529fc 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 53cdc696c823944f29b35d66c23609e9f36958e2 |
| Default Java | Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/9/testReport/ |
| Max. process+thread count | 544 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "cnauroth (via GitHub)" <gi...@apache.org>.
cnauroth commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2985515453
@adideshpande , @steveloughran and @ahmarsuhail , did this introduce a compilation error?
https://github.com/apache/hadoop/actions/runs/15741378616/job/44367291097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "adideshpande (via GitHub)" <gi...@apache.org>.
adideshpande commented on code in PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#discussion_r2082282938
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java:
##########
@@ -1339,6 +1340,37 @@ private Constants() {
public static final String AWS_SERVICE_IDENTIFIER_DDB = "DDB";
public static final String AWS_SERVICE_IDENTIFIER_STS = "STS";
+ /** Prefix for S3A client-specific properties.
+ * value: {@value}
+ */
+ public static final String FS_S3A_CLIENT_PREFIX = "fs.s3a.client.";
+
+ /** Custom headers postfix.
+ * value: {@value}
+ */
+ public static final String CUSTOM_HEADERS_POSTFIX = ".custom.headers";
+
+ /**
Review Comment:
Steve's comment was applicable here - https://github.com/apache/hadoop/pull/7379/files#diff-77c0bdf1ab74c4d2bc0f8b57d650039dd781e999220ee8a91bf2390f5a936db7R1356-R1364
and I have already addressed it by enclosing the comment within `<pre>... </pre>` blocks. Unfortunately, escaping literal with just `->` is not possible as the CI job throws a javadoc error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2846010438
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 53s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| -1 :x: | blanks | 0m 0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/5/artifact/out/blanks-eol.txt) | The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 48s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not generate ASF License warnings. |
| | | 73m 27s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7379 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 9130eff5e437 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7a3ac3faab98e2f7bdf5fbe9be2fb6e8bb72565c |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/5/testReport/ |
| Max. process+thread count | 558 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "adideshpande (via GitHub)" <gi...@apache.org>.
adideshpande commented on code in PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#discussion_r2098515721
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestAwsClientConfig.java:
##########
@@ -201,4 +208,64 @@ public void testCreateApiConnectionSettingsDefault() {
private void setOptionsToValue(String value, Configuration conf, String... keys) {
Arrays.stream(keys).forEach(key -> conf.set(key, value));
}
+
+ /**
+ * if {@link org.apache.hadoop.fs.s3a.Constants#CUSTOM_HEADERS_STS} is set,
+ * verify that returned client configuration has desired headers set.
+ */
+ @Test
+ public void testInitRequestHeadersForSTS() throws IOException {
+ final Configuration conf = new Configuration();
+ conf.set(CUSTOM_HEADERS_STS, "foo=bar;baz,qux=quux");
+ Assertions.assertThat(conf.get(CUSTOM_HEADERS_S3))
+ .describedAs("Custom client headers for s3 %s", CUSTOM_HEADERS_S3)
+ .isNull();
+
+ Assertions.assertThat(createClientConfigBuilder(conf, AWS_SERVICE_IDENTIFIER_S3)
+ .headers().size())
+ .describedAs("Count of S3 client headers")
+ .isEqualTo(0);
+ Assertions.assertThat(createClientConfigBuilder(conf, AWS_SERVICE_IDENTIFIER_STS)
+ .headers().size())
+ .describedAs("Count of STS client headers")
+ .isEqualTo(2);
+ Assertions.assertThat(createClientConfigBuilder(conf, AWS_SERVICE_IDENTIFIER_STS)
+ .headers().get("foo"))
+ .describedAs("STS client 'foo' header value")
+ .isEqualTo(Lists.newArrayList("bar", "baz"));
+ Assertions.assertThat(createClientConfigBuilder(conf, AWS_SERVICE_IDENTIFIER_STS)
+ .headers().get("qux"))
+ .describedAs("STS client 'qux' header value")
+ .isEqualTo(Lists.newArrayList("quux"));
+ }
+
+ /**
+ * if {@link org.apache.hadoop.fs.s3a.Constants#CUSTOM_HEADERS_S3} is set,
+ * verify that returned client configuration has desired headers set.
+ */
+ @Test
+ public void testInitRequestHeadersForS3() throws IOException {
+ final Configuration conf = new Configuration();
+ conf.set(CUSTOM_HEADERS_S3, "foo=bar;baz,qux=quux");
Review Comment:
Thanks for the suggestion. It's a good idea to have the header values trimmed and headers without any values should be skipped. So I have updated the logic and unit tests accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2985202077
merged, though I failed to give PradhanPrerak credit in the commit
@adideshpande can you a cherrypick and retest for branch-3.4; when we merge that I will expand the credit list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "ahmarsuhail (via GitHub)" <gi...@apache.org>.
ahmarsuhail commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2690249077
@adideshpande commit history is a bit messy, can you rebase your commit on top of trunk instead? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "ahmarsuhail (via GitHub)" <gi...@apache.org>.
ahmarsuhail commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2939660720
@adideshpande could you fix the checkstyle errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "adideshpande (via GitHub)" <gi...@apache.org>.
adideshpande commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2940482881
> @adideshpande could you fix the checkstyle errors.
Yes absolutely, fixed here - https://github.com/apache/hadoop/pull/7379/commits/74bfff3ff7394a00241452d142b63043e5133ebc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2940756798
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 9m 5s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 26s | | trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 0m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 26s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 0m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 38s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 49s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not generate ASF License warnings. |
| | | 89m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.50 ServerAPI=1.50 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7379 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux d041721a7551 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 74bfff3ff7394a00241452d142b63043e5133ebc |
| Default Java | Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/10/testReport/ |
| Max. process+thread count | 554 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran merged PR #7379:
URL: https://github.com/apache/hadoop/pull/7379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2867787498
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 21m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 49s | | trunk passed |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 0m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 1s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 0m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 20s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 40s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not generate ASF License warnings. |
| | | 103m 52s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7379 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux eca83285bb0f 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e84a89e4ffe249f278946f289579d7c413931c16 |
| Default Java | Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/7/testReport/ |
| Max. process+thread count | 554 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] [HADOOP-18562]: S3A: support custom S3 and STS headers [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7379:
URL: https://github.com/apache/hadoop/pull/7379#issuecomment-2688777674
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 16s | | https://github.com/apache/hadoop/pull/7379 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7379/2/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org