You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "saikatroy038 (via GitHub)" <gi...@apache.org> on 2025/01/27 12:17:06 UTC
[PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
saikatroy038 opened a new pull request, #7329:
URL: https://github.com/apache/hadoop/pull/7329
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Cherry-pick of https://github.com/apache/hadoop/pull/7011
### How was this patch tested?
integration tests ran on ap-south-1
Failing tests:
ITestAwsSdkWorkarounds.testNoisyLogging
ITestConnectionTimeouts.testObjectUploadTimeouts
ITestS3APutIfMatch.testPutIfAbsentConflict
ITestS3APutIfMatch.testPutIfAbsentLargeFileConflict
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2682578699
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +0 :ok: | mvndep | 5m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 14s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 22m 50s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 21m 49s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 40s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 2m 25s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 1m 52s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 28s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 50s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 40m 40s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 18m 16s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 18m 16s | | the patch passed |
| +1 :green_heart: | compile | 17m 19s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 17m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 32s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/11/artifact/out/results-checkstyle-root.txt) | root: The patch generated 274 new + 23 unchanged - 0 fixed = 297 total (was 23) |
| +1 :green_heart: | mvnsite | 2m 29s | | the patch passed |
| -1 :x: | javadoc | 1m 10s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/11/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 1m 31s | [/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/11/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 38m 17s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 36s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 56s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not generate ASF License warnings. |
| | | 261m 56s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Unread field:S3ABlockOutputStream.java:[line 1598] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9c9ebfecf71d 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / 883bc1db2185390201cbbe2e54276549f26c7a21 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/11/testReport/ |
| Max. process+thread count | 3140 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Support Conditional Writes in S3A [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2773797254
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 6m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 15s | | trunk passed |
| +1 :green_heart: | compile | 17m 49s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 5s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 59s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 7s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 36s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 36s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 15s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 19m 15s | [/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 19m 15s | [/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 15m 52s | [/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| -1 :x: | javac | 15m 52s | [/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 38s | [/buildtool-patch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 35s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | javadoc | 1m 7s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 35s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 33s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| -1 :x: | spotbugs | 0m 33s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 40m 18s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 39s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 32s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not generate ASF License warnings. |
| | | 243m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux a79ba509b3dd 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 75420278085b814cb06942163114c8b26f7f1c83 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/testReport/ |
| Max. process+thread count | 3136 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/26/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Support Conditional Writes in S3A [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2773904206
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 6m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 18s | | trunk passed |
| +1 :green_heart: | compile | 18m 26s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 18m 32s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 5m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 31s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 38m 59s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 16s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 16m 51s | [/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 16m 51s | [/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 15m 58s | [/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| -1 :x: | javac | 15m 58s | [/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 20s | [/buildtool-patch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 41s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | javadoc | 1m 10s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 41s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 37s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| -1 :x: | spotbugs | 0m 36s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | shadedclient | 39m 16s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 17s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 0m 33s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not generate ASF License warnings. |
| | | 243m 10s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux b608956bdcaa 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4b2dde03edc0d1adfcacc2d63a5b4d527d538870 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/testReport/ |
| Max. process+thread count | 1254 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/27/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Support Conditional Writes in S3A [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-3294414205
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2643307807
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +0 :ok: | mvndep | 6m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 11s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 18m 44s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 17m 23s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 4m 41s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 2m 34s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 2m 5s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 38s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 3m 51s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 39m 30s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 18m 15s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 18m 15s | | the patch passed |
| +1 :green_heart: | compile | 17m 10s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 17m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 26s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/artifact/out/results-checkstyle-root.txt) | root: The patch generated 133 new + 23 unchanged - 0 fixed = 156 total (was 23) |
| +1 :green_heart: | mvnsite | 2m 30s | | the patch passed |
| -1 :x: | javadoc | 0m 50s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) |
| -1 :x: | javadoc | 0m 46s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) |
| +1 :green_heart: | spotbugs | 4m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 42s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 38s | | hadoop-common in the patch passed. |
| -1 :x: | unit | 2m 55s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not generate ASF License warnings. |
| | | 252m 37s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.impl.TestRequestFactory |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 23f3f779377a 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / 26811e69a3212941067479a45daec69fc9ccf85a |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/testReport/ |
| Max. process+thread count | 2734 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2700018950
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 20s | | https://github.com/apache/hadoop/pull/7329 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/18/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2615909295
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 2s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 0m 42s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 36s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 33s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 0m 39s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 0m 41s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 9s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 37m 7s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 22s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 71 new + 5 unchanged - 0 fixed = 76 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 28s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| -1 :x: | javadoc | 0m 25s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 0s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 47s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 129m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 506c09f684c8 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / 29ff04acf62fa646368f04fd519d44f662996635 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/testReport/ |
| Max. process+thread count | 529 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2700519319
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +0 :ok: | mvndep | 6m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 55s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 20m 3s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 18m 7s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 33s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 2m 31s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 2m 5s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 14s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 40m 39s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 19m 23s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 19m 23s | | the patch passed |
| +1 :green_heart: | compile | 18m 58s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 18m 58s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 32s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/15/artifact/out/results-checkstyle-root.txt) | root: The patch generated 329 new + 23 unchanged - 0 fixed = 352 total (was 23) |
| +1 :green_heart: | mvnsite | 2m 29s | | the patch passed |
| -1 :x: | javadoc | 1m 12s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/15/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 1m 31s | [/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/15/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 38m 39s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 36s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 54s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not generate ASF License warnings. |
| | | 258m 26s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Unread field:S3ABlockOutputStream.java:[line 1609] |
| | Unread field:S3ABlockOutputStream.java:[line 1603] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 067cfe6c4e5a 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / 4423ab0f22083acddbcc6873a2320ee5cc311a11 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/15/testReport/ |
| Max. process+thread count | 1256 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/15/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Support Conditional Writes in S3A [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2773762550
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 6m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 26s | | trunk passed |
| +1 :green_heart: | compile | 8m 23s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 7m 18s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 14s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 29s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 8m 10s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 8m 10s | | the patch passed |
| +1 :green_heart: | compile | 8m 18s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 8m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 2m 15s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/28/artifact/out/results-checkstyle-root.txt) | root: The patch generated 52 new + 23 unchanged - 0 fixed = 75 total (was 23) |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| -1 :x: | javadoc | 0m 36s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/28/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 14s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 12m 28s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 58s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 135m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/28/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux c2db4139accd 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 210c66858017600676bb28dd6805e31b0ecb581f |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/28/testReport/ |
| Max. process+thread count | 1261 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/28/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2639812172
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 46s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 0m 45s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 34s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 0m 41s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 0m 40s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 9s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 38m 17s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 65 new + 5 unchanged - 0 fixed = 70 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| -1 :x: | javadoc | 0m 29s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| -1 :x: | javadoc | 0m 26s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 54s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 45s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 132m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3212193e2d22 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / b19da55cc3664c6145d8285e2e8643c938036118 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/testReport/ |
| Max. process+thread count | 528 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2666118030
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +0 :ok: | mvndep | 5m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 52s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 18m 43s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 17m 12s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 33s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 2m 30s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 2m 6s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 38s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 50s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 38m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 17m 42s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 17m 42s | | the patch passed |
| +1 :green_heart: | compile | 17m 14s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 17m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 25s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/artifact/out/results-checkstyle-root.txt) | root: The patch generated 244 new + 23 unchanged - 0 fixed = 267 total (was 23) |
| +1 :green_heart: | mvnsite | 2m 29s | | the patch passed |
| -1 :x: | javadoc | 1m 10s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 50s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 45s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| -1 :x: | spotbugs | 1m 30s | [/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 38m 53s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 40s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 59s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not generate ASF License warnings. |
| | | 250m 6s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Unread field:S3ABlockOutputStream.java:[line 1598] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux aa90e18fae55 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / 294b14d1fb9bb2d51ece945efbe2704efb0b73b6 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/testReport/ |
| Max. process+thread count | 1255 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2698513979
@saikatroy038
```
Another iteration;
* pulled out *all* the stuff related to the notion of a configuration service -needs to be independent PR
* "simplified" logic as to when to use conditional create for a normal create() call, even without any explicit request.
This comes down to: if you are using the performance flag for creation, or when you create an fs with `create` in `fs.s3a.performance.flags` then we switch to using condition create if:
overwrite=false and conditional create is available and the create is not a conditional etag creation.
---
Somehow those conditional options from the builder of S3ABlockOutput stream aren't getting in.
I actually realise these are duplicate; putting them into the `PutObjectOptions` settings is enough
in S3AFileSystem, L2181, the PutObjectOptions created must set the etag/conditional create flag
from that calculated in the `innerCreateFile`
## Tests.
### Performance flag && !overwrite triggers conditional create
Create a file with
```
opt("fs.s3a.create.performance",true)
.overwrite(false)
```
and verify that this is mapped to a conditional create
### Behaviour when FS doesn't support conditional create
Add a new test suite which sets
FS_S3A_CONDITIONAL_CREATE_ENABLED=true to the
FS creation configuration, then verify that the request is not
issued with conditional options
1. `opt(conditional create, true) => PathIOException`
2. `opt(conditional create etag, "string") => PathIOException`
3. `opt("fs.s3a.create.performance",true).overwrite(false)` succeeds with no conditional IO attempted
This test suite should not be skipped if conditional IO is disabled,
because it is the behaviour we require there.
### existing tests
`testIfMatchCreateFileWithoutOverwriteWithPerformanceFlag`
You can't set filesystem configuration options once the FS has been created...you'll have to work out a different solution.
Look at `ITestCreateFileCost.testCreateFilePerformanceFlag()` to see how to do it, and how the costs tests have an easy way to make assertions about statistics.
### everywhere
Add assertions for the open input streams to .haveCapability() of the conditional create and etag flags, into existing tests, including those where the performance flag is expected to turn this on.
Why? this allows us to make fast asserts that the relevant flag has been passed down, without even waiting for a close() to complete and then asserting on the statistics (these are still important, but a fast failure helps isolate issues to flag propagation, rather than statistic collection.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Support Conditional Writes in S3A [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2774125940
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 7m 8s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 15s | | trunk passed |
| +1 :green_heart: | compile | 19m 26s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 15m 27s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 13s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 40m 41s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 1s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 16m 42s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 16m 42s | | the patch passed |
| +1 :green_heart: | compile | 15m 25s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 15m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 34s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/29/artifact/out/results-checkstyle-root.txt) | root: The patch generated 52 new + 23 unchanged - 0 fixed = 75 total (was 23) |
| +1 :green_heart: | mvnsite | 2m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 0s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 4m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 14s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 14m 51s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 3m 35s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not generate ASF License warnings. |
| | | 245m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/29/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux fe2a8a87d2a8 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6fc0203c1b2389f5e9f4566a94aaba3bc132ff5d |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/29/testReport/ |
| Max. process+thread count | 3136 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/29/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2667806701
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +0 :ok: | mvndep | 6m 34s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 47m 43s | [/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-mvninstall-root.txt) | root in HADOOP-19256-s3-conditional-writes failed. |
| -1 :x: | compile | 0m 31s | [/branch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | root in HADOOP-19256-s3-conditional-writes failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 0m 31s | [/branch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | root in HADOOP-19256-s3-conditional-writes failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| +1 :green_heart: | checkstyle | 5m 47s | | HADOOP-19256-s3-conditional-writes passed |
| -1 :x: | mvnsite | 0m 33s | [/branch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in HADOOP-19256-s3-conditional-writes failed. |
| -1 :x: | mvnsite | 0m 42s | [/branch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in HADOOP-19256-s3-conditional-writes failed. |
| -1 :x: | javadoc | 0m 31s | [/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in HADOOP-19256-s3-conditional-writes failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 31s | [/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in HADOOP-19256-s3-conditional-writes failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 31s | [/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-common in HADOOP-19256-s3-conditional-writes failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| -1 :x: | spotbugs | 1m 39s | [/branch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common in HADOOP-19256-s3-conditional-writes failed. |
| -1 :x: | spotbugs | 0m 31s | [/branch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/branch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in HADOOP-19256-s3-conditional-writes failed. |
| +1 :green_heart: | shadedclient | 13m 3s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 44s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 49s | [/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 0m 30s | [/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | compile | 1m 20s | [/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 1m 20s | [/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-compile-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 0m 13s | [/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| -1 :x: | javac | 0m 13s | [/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 14s | [/buildtool-patch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/buildtool-patch-checkstyle-root.txt) | The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 55s | [/patch-mvnsite-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | mvnsite | 0m 24s | [/patch-mvnsite-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-mvnsite-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | javadoc | 1m 10s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 26s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 35s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 generated 24 new + 0 unchanged - 0 fixed = 24 total (was 0) |
| -1 :x: | spotbugs | 0m 24s | [/patch-spotbugs-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-spotbugs-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | spotbugs | 0m 24s | [/patch-spotbugs-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-spotbugs-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | shadedclient | 16m 45s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 27s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| -1 :x: | unit | 0m 32s | [/patch-unit-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt) | hadoop-aws in the patch failed. |
| -1 :x: | asflicense | 0m 44s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 88m 22s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f7f7dd227502 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / b00766488aad36d5407e35b215993cffd9208f77 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/testReport/ |
| Max. process+thread count | 84 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/7/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "saikatroy038 (via GitHub)" <gi...@apache.org>.
saikatroy038 commented on code in PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#discussion_r1944538668
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/S3MagicCommitTracker.java:
##########
@@ -79,7 +79,7 @@ public boolean aboutToComplete(String uploadId,
PutObjectRequest originalDestPut = getWriter().createPutObjectRequest(
getOriginalDestKey(),
0,
- new PutObjectOptions(null, headers));
+ new PutObjectOptions(true,false, null, headers));
Review Comment:
done
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java:
##########
@@ -62,6 +62,7 @@
import static org.apache.commons.lang3.StringUtils.isNotEmpty;
import static org.apache.hadoop.fs.s3a.Constants.DEFAULT_PART_UPLOAD_TIMEOUT;
+import static org.apache.hadoop.fs.s3a.impl.AWSHeaders.IF_NONE_MATCH;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "saikatroy038 (via GitHub)" <gi...@apache.org>.
saikatroy038 commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2700862056
with `skipHead` and `skipList` in `S3AFileSystem.innerCreateFile()`, the file will be overridden if etag is passed.. even if we set `fsBuilder.overwrite(false)`. is that expected behaviour?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "saikatroy038 (via GitHub)" <gi...@apache.org>.
saikatroy038 commented on code in PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#discussion_r1944538173
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -101,24 +99,26 @@ private static void createFileWithIfNoneMatchFlag(
Path path,
byte[] data,
String ifMatchTag) throws Exception {
- FSDataOutputStreamBuilder builder = fs.createFile(path);
- builder.must(FS_S3A_CONDITIONAL_FILE_CREATE, ifMatchTag);
- FSDataOutputStream stream = builder.create().build();
- if (data != null && data.length > 0) {
- stream.write(data);
- }
- stream.close();
- IOUtils.closeStream(stream);
+ FSDataOutputStreamBuilder builder = fs.createFile(path);
+ builder.must(FS_S3A_CONDITIONAL_FILE_CREATE, "true");
+ builder.opt(FS_S3A_CREATE_HEADER + "." + IF_NONE_MATCH, ifMatchTag);
+ FSDataOutputStream stream = builder.create().build();
+ if (data != null && data.length > 0) {
+ stream.write(data);
+ }
+ stream.close();
+ IOUtils.closeStream(stream);
}
@Test
public void testPutIfAbsentConflict() throws Throwable {
FileSystem fs = getFileSystem();
Path testFile = methodPath();
-
fs.mkdirs(testFile.getParent());
byte[] fileBytes = dataset(TEST_FILE_LEN, 0, 255);
+ createFileWithIfNoneMatchFlag(fs, testFile, fileBytes, "*");
+
RemoteFileChangedException firstException = intercept(RemoteFileChangedException.class,
() -> createFileWithIfNoneMatchFlag(fs, testFile, fileBytes, "*"));
assertS3ExceptionStatusCode(412, firstException);
Review Comment:
done
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+import org.junit.Test;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import java.io.IOException;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CONDITIONAL_FILE_CREATE;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_HEADER;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_MIN_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.AWSHeaders.IF_NONE_MATCH;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1MB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private Configuration conf;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2704044505
good question. options
1. overwrite takes priority. Good; consistent with the rest. Bad: people explicitly setting etag may be surprised
1. etag priority. Good: resolves ambiguity about etag flag.
1. reject as an illegal state.
for 1 and 2 we log at debug too, obviously
I think 1 or 3 is correct, with 3 being the strict one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2682166244
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +0 :ok: | mvndep | 6m 6s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 16s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 18m 52s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 17m 14s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 30s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 2m 31s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 2m 4s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 37s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 51s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 38m 42s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 19m 22s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 19m 22s | | the patch passed |
| +1 :green_heart: | compile | 19m 40s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 19m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 32s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/10/artifact/out/results-checkstyle-root.txt) | root: The patch generated 266 new + 23 unchanged - 0 fixed = 289 total (was 23) |
| +1 :green_heart: | mvnsite | 5m 15s | | the patch passed |
| -1 :x: | javadoc | 1m 9s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/10/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 1m 33s | [/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/10/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 40m 44s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 24s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 58s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not generate ASF License warnings. |
| | | 260m 7s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Unread field:S3ABlockOutputStream.java:[line 1598] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5f43868e11e3 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / 6a8e199c6e0b5238d2bf5b6c9f7f6196a6bce22a |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/10/testReport/ |
| Max. process+thread count | 2140 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/10/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2616407569
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 52s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 0m 44s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 32s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 0m 41s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 0m 41s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 8s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 38m 52s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-aws.txt) | hadoop-tools/hadoop-aws: The patch generated 70 new + 5 unchanged - 0 fixed = 75 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| -1 :x: | javadoc | 0m 29s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-tools_hadoop-aws-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| -1 :x: | javadoc | 0m 25s | [/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/artifact/out/results-javadoc-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga.txt) | hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 38s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 44s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 131m 25s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 88b6aaed7d3f 5.15.0-125-generic #135-Ubuntu SMP Fri Sep 27 13:53:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / aa1defccf49717b7be391ebde37d0346fd8bb921 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/testReport/ |
| Max. process+thread count | 641 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2676805905
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 18m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +0 :ok: | mvndep | 6m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 10s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 18m 50s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 17m 17s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 39s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 2m 32s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 2m 4s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 36s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 50s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 38m 27s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 18m 34s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 18m 34s | | the patch passed |
| +1 :green_heart: | compile | 17m 16s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 17m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 31s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/artifact/out/results-checkstyle-root.txt) | root: The patch generated 244 new + 23 unchanged - 0 fixed = 267 total (was 23) |
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| -1 :x: | javadoc | 1m 11s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 1m 31s | [/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 38m 27s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 35s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 55s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not generate ASF License warnings. |
| | | 269m 8s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Unread field:S3ABlockOutputStream.java:[line 1598] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 23195b073f37 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / b00766488aad36d5407e35b215993cffd9208f77 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/testReport/ |
| Max. process+thread count | 2927 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/9/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Support Conditional Writes in S3A [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #7329: HADOOP-19256 Support Conditional Writes in S3A
URL: https://github.com/apache/hadoop/pull/7329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#discussion_r1932400430
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+import org.junit.Test;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import java.io.IOException;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CONDITIONAL_FILE_CREATE;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_HEADER;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_MIN_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.AWSHeaders.IF_NONE_MATCH;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1MB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private Configuration conf;
+
+ @Override
+ public Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+ S3ATestUtils.disableFilesystemCaching(conf);
+ removeBaseAndBucketOverrides(
+ conf,
+ MULTIPART_SIZE,
+ UPLOAD_PART_COUNT_LIMIT,
+ MIN_MULTIPART_THRESHOLD);
+ conf.setLong(UPLOAD_PART_COUNT_LIMIT, 2);
+ conf.setLong(MIN_MULTIPART_THRESHOLD, MULTIPART_MIN_SIZE);
+ conf.setInt(MULTIPART_SIZE, MULTIPART_MIN_SIZE);
+ return conf;
+ }
+
+ @Override
+ public void setup() throws Exception {
+ super.setup();
+ conf = createConfiguration();
Review Comment:
not needed as `super.setup()` creates the config. Instead use
```
Configuration conf = getConfiguration();
```
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -101,24 +99,26 @@ private static void createFileWithIfNoneMatchFlag(
Path path,
byte[] data,
String ifMatchTag) throws Exception {
- FSDataOutputStreamBuilder builder = fs.createFile(path);
- builder.must(FS_S3A_CONDITIONAL_FILE_CREATE, ifMatchTag);
- FSDataOutputStream stream = builder.create().build();
- if (data != null && data.length > 0) {
- stream.write(data);
- }
- stream.close();
- IOUtils.closeStream(stream);
+ FSDataOutputStreamBuilder builder = fs.createFile(path);
+ builder.must(FS_S3A_CONDITIONAL_FILE_CREATE, "true");
+ builder.opt(FS_S3A_CREATE_HEADER + "." + IF_NONE_MATCH, ifMatchTag);
+ FSDataOutputStream stream = builder.create().build();
+ if (data != null && data.length > 0) {
+ stream.write(data);
+ }
+ stream.close();
+ IOUtils.closeStream(stream);
}
@Test
public void testPutIfAbsentConflict() throws Throwable {
FileSystem fs = getFileSystem();
Path testFile = methodPath();
-
fs.mkdirs(testFile.getParent());
byte[] fileBytes = dataset(TEST_FILE_LEN, 0, 255);
+ createFileWithIfNoneMatchFlag(fs, testFile, fileBytes, "*");
+
RemoteFileChangedException firstException = intercept(RemoteFileChangedException.class,
() -> createFileWithIfNoneMatchFlag(fs, testFile, fileBytes, "*"));
assertS3ExceptionStatusCode(412, firstException);
Review Comment:
we've got a constant in org.apache.hadoop.fs.s3a.impl.InternalConstants for this number
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+import org.junit.Test;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import java.io.IOException;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CONDITIONAL_FILE_CREATE;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_HEADER;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_MIN_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.AWSHeaders.IF_NONE_MATCH;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1MB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private Configuration conf;
Review Comment:
cut
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import org.apache.hadoop.conf.Configuration;
Review Comment:
import ordering is wrong...i'll give you the IDE settings for this
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java:
##########
@@ -62,6 +62,7 @@
import static org.apache.commons.lang3.StringUtils.isNotEmpty;
import static org.apache.hadoop.fs.s3a.Constants.DEFAULT_PART_UPLOAD_TIMEOUT;
+import static org.apache.hadoop.fs.s3a.impl.AWSHeaders.IF_NONE_MATCH;
Review Comment:
move below line 67
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -101,24 +99,26 @@ private static void createFileWithIfNoneMatchFlag(
Path path,
byte[] data,
String ifMatchTag) throws Exception {
- FSDataOutputStreamBuilder builder = fs.createFile(path);
- builder.must(FS_S3A_CONDITIONAL_FILE_CREATE, ifMatchTag);
- FSDataOutputStream stream = builder.create().build();
- if (data != null && data.length > 0) {
- stream.write(data);
- }
- stream.close();
- IOUtils.closeStream(stream);
+ FSDataOutputStreamBuilder builder = fs.createFile(path);
+ builder.must(FS_S3A_CONDITIONAL_FILE_CREATE, "true");
+ builder.opt(FS_S3A_CREATE_HEADER + "." + IF_NONE_MATCH, ifMatchTag);
+ FSDataOutputStream stream = builder.create().build();
Review Comment:
use try-with-resources here.
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/S3MagicCommitTracker.java:
##########
@@ -79,7 +79,7 @@ public boolean aboutToComplete(String uploadId,
PutObjectRequest originalDestPut = getWriter().createPutObjectRequest(
getOriginalDestKey(),
0,
- new PutObjectOptions(null, headers));
+ new PutObjectOptions(true,false, null, headers));
Review Comment:
style nit: put a space after the comma. checkstyle will complain about this
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/PutObjectOptions.java:
##########
@@ -26,6 +26,16 @@
*/
public final class PutObjectOptions {
+ /**
+ * Can the PUT operation skip marker deletion?
+ */
+ private final boolean keepMarkers;
+
+ /**
+ * Is this a conditional PUT operation
+ */
+ private final boolean conditionalPutEnabled;
Review Comment:
given there's now a etag match too, maybe we should give this a name, such as
`noObjectOverwrite`
we can then add a new one after `etagOverwrite` which will be a string
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java:
##########
@@ -533,12 +534,19 @@ public CreateMultipartUploadRequest.Builder newMultipartUploadRequestBuilder(
public CompleteMultipartUploadRequest.Builder newCompleteMultipartUploadRequestBuilder(
String destKey,
String uploadId,
- List<CompletedPart> partETags) {
+ List<CompletedPart> partETags,
+ PutObjectOptions putOptions) {
+
// a copy of the list is required, so that the AWS SDK doesn't
// attempt to sort an unmodifiable list.
- CompleteMultipartUploadRequest.Builder requestBuilder =
- CompleteMultipartUploadRequest.builder().bucket(bucket).key(destKey).uploadId(uploadId)
+ CompleteMultipartUploadRequest.Builder requestBuilder;
+ Map<String, String> optionHeaders = putOptions.getHeaders();
+ requestBuilder = CompleteMultipartUploadRequest.builder().bucket(bucket).key(destKey).uploadId(uploadId)
.multipartUpload(CompletedMultipartUpload.builder().parts(partETags).build());
+ if (putOptions.isconditionalPutEnabled()) {
+ requestBuilder.overrideConfiguration(
Review Comment:
this is from the first patch, isn't it? I'll have to look at my comments there
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/PutObjectOptions.java:
##########
@@ -26,6 +26,16 @@
*/
public final class PutObjectOptions {
+ /**
+ * Can the PUT operation skip marker deletion?
+ */
+ private final boolean keepMarkers;
+
+ /**
+ * Is this a conditional PUT operation
Review Comment:
add a ? at the end
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+import org.junit.Test;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import java.io.IOException;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CONDITIONAL_FILE_CREATE;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_HEADER;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_MIN_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.AWSHeaders.IF_NONE_MATCH;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1MB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private Configuration conf;
+
+ @Override
+ public Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+ S3ATestUtils.disableFilesystemCaching(conf);
+ removeBaseAndBucketOverrides(
+ conf,
+ MULTIPART_SIZE,
+ UPLOAD_PART_COUNT_LIMIT,
+ MIN_MULTIPART_THRESHOLD);
+ conf.setLong(UPLOAD_PART_COUNT_LIMIT, 2);
+ conf.setLong(MIN_MULTIPART_THRESHOLD, MULTIPART_MIN_SIZE);
+ conf.setInt(MULTIPART_SIZE, MULTIPART_MIN_SIZE);
+ return conf;
+ }
+
+ @Override
+ public void setup() throws Exception {
+ super.setup();
+ conf = createConfiguration();
+ skipIfNotEnabled(conf, FS_S3A_CONDITIONAL_FILE_CREATE,
+ "Skipping IfNoneMatch tests");
+ }
+
+ private static void assertS3ExceptionStatusCode(int code, Exception ex) {
+ S3Exception s3Exception = (S3Exception) ex.getCause();
+
+ if (s3Exception.statusCode() != code) {
+ throw new AssertionError("Expected status code " + code + " from " + ex, ex);
+ }
+ }
+
+ protected String getBlockOutputBufferName() {
+ return FAST_UPLOAD_BUFFER_ARRAY;
+ }
+
+ /**
+ * Create a file using the PutIfMatch feature from S3
+ * @param fs filesystem
+ * @param path path to write
+ * @param data source dataset. Can be null
+ * @throws IOException on any problem
+ */
+ private static void createFileWithIfNoneMatchFlag(
+ FileSystem fs,
+ Path path,
+ byte[] data,
+ String ifMatchTag) throws Exception {
+ FSDataOutputStreamBuilder builder = fs.createFile(path);
+ builder.must(FS_S3A_CONDITIONAL_FILE_CREATE, "true");
+ builder.opt(FS_S3A_CREATE_HEADER + "." + IF_NONE_MATCH, ifMatchTag);
+ FSDataOutputStream stream = builder.create().build();
+ if (data != null && data.length > 0) {
+ stream.write(data);
+ }
+ stream.close();
+ IOUtils.closeStream(stream);
+ }
+
+ @Test
+ public void testPutIfAbsentConflict() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+ byte[] fileBytes = dataset(TEST_FILE_LEN, 0, 255);
+
+ createFileWithIfNoneMatchFlag(fs, testFile, fileBytes, "*");
+
+ RemoteFileChangedException firstException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithIfNoneMatchFlag(fs, testFile, fileBytes, "*"));
+ assertS3ExceptionStatusCode(412, firstException);
+
+ RemoteFileChangedException secondException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithIfNoneMatchFlag(fs, testFile, fileBytes, "*"));
+ assertS3ExceptionStatusCode(412, secondException);
+ }
+
+
+ @Test
+ public void testPutIfAbsentLargeFileConflict() throws Throwable {
Review Comment:
I'd offered a way to test this without needing to create a large file for a multipart upload...we can work on this...anything uploading 6 MB gets moved into the scale tests, so doesn't get run as often
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/PutObjectOptions.java:
##########
@@ -39,15 +49,36 @@ public final class PutObjectOptions {
/**
* Constructor.
* @param storageClass Storage class, if not null.
+ * @param conditionalPutEnabled Is this a conditional Put?
* @param headers Headers; may be null.
*/
public PutObjectOptions(
+ final boolean keepMarkers,
Review Comment:
the keepmarkers option is something I've cut from trunk. It's come in here during rebase merge work.
proposed;
* you move the new argument to after the `headers` entry
* remove the keepmarkers argument/field and all uses of it in this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "saikatroy038 (via GitHub)" <gi...@apache.org>.
saikatroy038 commented on code in PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#discussion_r1944539120
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/PutObjectOptions.java:
##########
@@ -39,15 +49,36 @@ public final class PutObjectOptions {
/**
* Constructor.
* @param storageClass Storage class, if not null.
+ * @param conditionalPutEnabled Is this a conditional Put?
* @param headers Headers; may be null.
*/
public PutObjectOptions(
+ final boolean keepMarkers,
Review Comment:
done
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.io.IOUtils;
+
+import org.junit.Test;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import java.io.IOException;
+
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CONDITIONAL_FILE_CREATE;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_HEADER;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_MIN_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.impl.AWSHeaders.IF_NONE_MATCH;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1MB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private Configuration conf;
+
+ @Override
+ public Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+ S3ATestUtils.disableFilesystemCaching(conf);
+ removeBaseAndBucketOverrides(
+ conf,
+ MULTIPART_SIZE,
+ UPLOAD_PART_COUNT_LIMIT,
+ MIN_MULTIPART_THRESHOLD);
+ conf.setLong(UPLOAD_PART_COUNT_LIMIT, 2);
+ conf.setLong(MIN_MULTIPART_THRESHOLD, MULTIPART_MIN_SIZE);
+ conf.setInt(MULTIPART_SIZE, MULTIPART_MIN_SIZE);
+ return conf;
+ }
+
+ @Override
+ public void setup() throws Exception {
+ super.setup();
+ conf = createConfiguration();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2682607096
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 7m 17s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +0 :ok: | mvndep | 5m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 17m 57s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 8m 32s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 7m 46s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 54s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 1m 28s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 1m 21s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 5s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 13s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 22m 44s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 8m 37s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 8m 37s | | the patch passed |
| +1 :green_heart: | compile | 8m 22s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 8m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 54s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/12/artifact/out/results-checkstyle-root.txt) | root: The patch generated 275 new + 23 unchanged - 0 fixed = 298 total (was 23) |
| +1 :green_heart: | mvnsite | 1m 30s | | the patch passed |
| -1 :x: | javadoc | 0m 42s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/12/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 0m 58s | [/new-spotbugs-hadoop-tools_hadoop-aws.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/12/artifact/out/new-spotbugs-hadoop-tools_hadoop-aws.html) | hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 20m 8s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 16m 49s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 17s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not generate ASF License warnings. |
| | | 146m 59s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-aws |
| | Unread field:S3ABlockOutputStream.java:[line 1609] |
| | Unread field:S3ABlockOutputStream.java:[line 1603] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8cf9a31cfabb 5.15.0-130-generic #140-Ubuntu SMP Wed Dec 18 17:59:53 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / 9e5ff77889372d89a42ce2fe43aebad56236912b |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/12/testReport/ |
| Max. process+thread count | 3150 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/12/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on code in PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#discussion_r1977939528
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,564 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.FileNotFoundException;
+import java.nio.charset.StandardCharsets;
+
+import org.junit.Assume;
+import org.junit.Test;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileAlreadyExistsException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3AFileStatus;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.fs.s3a.Statistic;
+import org.apache.hadoop.fs.s3a.statistics.BlockOutputStreamStatistics;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE;
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_MULTIPART;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_OVERWRITE_SUPPORTED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_404_NOT_FOUND;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_412_PRECONDITION_FAILED;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1KB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private static final int UPDATED_MULTIPART_THRESHOLD = 100 * _1KB;
+
+ private static final byte[] SMALL_FILE_BYTES = dataset(TEST_FILE_LEN, 0, 255);
+ private static final byte[] MULTIPART_FILE_BYTES = dataset(UPDATED_MULTIPART_THRESHOLD * 5, 'a', 'z' - 'a');
+
+ private BlockOutputStreamStatistics statistics;
+
+ @Override
+ public Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+
+ S3ATestUtils.disableFilesystemCaching(conf);
+ removeBaseAndBucketOverrides(
+ conf,
+ MULTIPART_SIZE,
+ UPLOAD_PART_COUNT_LIMIT,
+ MIN_MULTIPART_THRESHOLD);
+ conf.setLong(UPLOAD_PART_COUNT_LIMIT, 2);
+ conf.setLong(MIN_MULTIPART_THRESHOLD, UPDATED_MULTIPART_THRESHOLD);
+ conf.setInt(MULTIPART_SIZE, UPDATED_MULTIPART_THRESHOLD);
+ return conf;
+ }
+
+ @Override
+ public void setup() throws Exception {
+ super.setup();
+ Configuration conf = getConfiguration();
+ skipIfNotEnabled(conf, FS_S3A_CREATE_OVERWRITE_SUPPORTED,
+ "Skipping IfNoneMatch tests");
+ }
+
+ private static void assertS3ExceptionStatusCode(int code, Exception ex) {
+ S3Exception s3Exception = (S3Exception) ex.getCause();
+
+ if (s3Exception.statusCode() != code) {
+ throw new AssertionError("Expected status code " + code + " from " + ex, ex);
+ }
+ }
+
+ protected String getBlockOutputBufferName() {
+ return FAST_UPLOAD_BUFFER_ARRAY;
+ }
+
+ private static void createFileWithFlags(
+ FileSystem fs,
+ Path path,
+ byte[] data,
+ boolean ifNoneMatchFlag,
+ String etag,
+ boolean forceMultipart) throws Exception {
+ FSDataOutputStream stream = getStreamWithFlags(fs, path, ifNoneMatchFlag, etag, forceMultipart);
+ if (data != null && data.length > 0) {
+ stream.write(data);
+ }
+ stream.close();
+ }
+
+ private static void createFileWithFlags(
+ FileSystem fs,
+ Path path,
+ byte[] data,
+ boolean ifNoneMatchFlag,
+ String etag) throws Exception {
+ createFileWithFlags(fs, path, data, ifNoneMatchFlag, etag, false);
+ }
+
+ private static FSDataOutputStream getStreamWithFlags(
Review Comment:
add javadocs for all these methods...test code needs to be as rigorously documented as production code
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,564 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.FileNotFoundException;
Review Comment:
mport layout needs tuning, the desired layout should be
java.*
javax.*
everything not in org.apache
org.apache.*, alphabetical order
static import.*
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,564 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.FileNotFoundException;
+import java.nio.charset.StandardCharsets;
+
+import org.junit.Assume;
+import org.junit.Test;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileAlreadyExistsException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3AFileStatus;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.fs.s3a.Statistic;
+import org.apache.hadoop.fs.s3a.statistics.BlockOutputStreamStatistics;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE;
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_MULTIPART;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_OVERWRITE_SUPPORTED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_404_NOT_FOUND;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_412_PRECONDITION_FAILED;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1KB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private static final int UPDATED_MULTIPART_THRESHOLD = 100 * _1KB;
+
+ private static final byte[] SMALL_FILE_BYTES = dataset(TEST_FILE_LEN, 0, 255);
+ private static final byte[] MULTIPART_FILE_BYTES = dataset(UPDATED_MULTIPART_THRESHOLD * 5, 'a', 'z' - 'a');
+
+ private BlockOutputStreamStatistics statistics;
+
+ @Override
+ public Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
Review Comment:
wrong indentation; hadoop uses two spaces per indentation, not four.
Change your IDE settings then do a whole-file reformat
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,564 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.FileNotFoundException;
+import java.nio.charset.StandardCharsets;
+
+import org.junit.Assume;
+import org.junit.Test;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileAlreadyExistsException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3AFileStatus;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.fs.s3a.Statistic;
+import org.apache.hadoop.fs.s3a.statistics.BlockOutputStreamStatistics;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE;
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_MULTIPART;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_OVERWRITE_SUPPORTED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_404_NOT_FOUND;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_412_PRECONDITION_FAILED;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1KB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private static final int UPDATED_MULTIPART_THRESHOLD = 100 * _1KB;
+
+ private static final byte[] SMALL_FILE_BYTES = dataset(TEST_FILE_LEN, 0, 255);
+ private static final byte[] MULTIPART_FILE_BYTES = dataset(UPDATED_MULTIPART_THRESHOLD * 5, 'a', 'z' - 'a');
+
+ private BlockOutputStreamStatistics statistics;
+
+ @Override
+ public Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+
+ S3ATestUtils.disableFilesystemCaching(conf);
+ removeBaseAndBucketOverrides(
+ conf,
+ MULTIPART_SIZE,
+ UPLOAD_PART_COUNT_LIMIT,
+ MIN_MULTIPART_THRESHOLD);
+ conf.setLong(UPLOAD_PART_COUNT_LIMIT, 2);
+ conf.setLong(MIN_MULTIPART_THRESHOLD, UPDATED_MULTIPART_THRESHOLD);
+ conf.setInt(MULTIPART_SIZE, UPDATED_MULTIPART_THRESHOLD);
+ return conf;
+ }
+
+ @Override
+ public void setup() throws Exception {
+ super.setup();
+ Configuration conf = getConfiguration();
+ skipIfNotEnabled(conf, FS_S3A_CREATE_OVERWRITE_SUPPORTED,
+ "Skipping IfNoneMatch tests");
+ }
+
+ private static void assertS3ExceptionStatusCode(int code, Exception ex) {
+ S3Exception s3Exception = (S3Exception) ex.getCause();
+
+ if (s3Exception.statusCode() != code) {
+ throw new AssertionError("Expected status code " + code + " from " + ex, ex);
+ }
+ }
+
+ protected String getBlockOutputBufferName() {
+ return FAST_UPLOAD_BUFFER_ARRAY;
+ }
+
+ private static void createFileWithFlags(
+ FileSystem fs,
+ Path path,
+ byte[] data,
+ boolean ifNoneMatchFlag,
+ String etag,
+ boolean forceMultipart) throws Exception {
+ FSDataOutputStream stream = getStreamWithFlags(fs, path, ifNoneMatchFlag, etag, forceMultipart);
+ if (data != null && data.length > 0) {
+ stream.write(data);
+ }
+ stream.close();
+ }
+
+ private static void createFileWithFlags(
+ FileSystem fs,
+ Path path,
+ byte[] data,
+ boolean ifNoneMatchFlag,
+ String etag) throws Exception {
+ createFileWithFlags(fs, path, data, ifNoneMatchFlag, etag, false);
+ }
+
+ private static FSDataOutputStream getStreamWithFlags(
+ FileSystem fs,
+ Path path,
+ boolean ifNoneMatchFlag,
+ String etag,
+ boolean forceMultipart) throws Exception {
+ FSDataOutputStreamBuilder builder = fs.createFile(path);
+ if (ifNoneMatchFlag) {
+ builder.must(FS_OPTION_CREATE_CONDITIONAL_OVERWRITE, "true");
+ }
+ if (etag != null) {
+ builder.must(FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG, etag);
+ }
+ if (forceMultipart) {
+ builder.opt(FS_S3A_CREATE_MULTIPART, "true");
+ }
+ return builder.create().build();
+ }
+
+ private static FSDataOutputStream getStreamWithFlags(
+ FileSystem fs,
+ Path path,
+ boolean ifNoneMatchFlag,
+ String etag) throws Exception {
+ return getStreamWithFlags(fs, path, ifNoneMatchFlag, etag, false);
+ }
+
+ private static String readFileContent(FileSystem fs, Path path) throws Throwable {
+ try (FSDataInputStream inputStream = fs.open(path)) {
+ return IOUtils.toString(inputStream, StandardCharsets.UTF_8);
+ }
+ }
+
+ private void updateStatistics(FSDataOutputStream stream) {
+ statistics = S3ATestUtils.getOutputStreamStatistics(stream);
+ }
+
+ @Test
+ public void testIfNoneMatchConflictOnOverwrite() throws Throwable {
+ describe("generate conflict on overwrites");
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create a file over an empty path: all good
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // attempted overwrite fails
+ RemoteFileChangedException firstException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, firstException);
+
+ // second attempt also fails
+ RemoteFileChangedException secondException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, secondException);
+
+ // Delete file and verify an overwrite works again
+ fs.delete(testFile, false);
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+ }
+
+ @Test
+ public void testIfNoneMatchConflictOnMultipartUpload() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip if multipart upload not supported
+ Assume.assumeTrue("Skipping as multipart upload not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true);
+
+ RemoteFileChangedException firstException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, firstException);
+
+ RemoteFileChangedException secondException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, secondException);
+ }
+
+ @Test
+ public void testIfNoneMatchMultipartUploadWithRaceCondition() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip test if multipart uploads are not supported
+ Assume.assumeTrue("Skipping test as multipart uploads are not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ // Create a file with multipart upload but do not close the stream
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null, true);
+ stream.write(MULTIPART_FILE_BYTES);
+
+ // create and close another small file in parallel
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // Closing the first stream should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchTwoConcurrentMultipartUploads() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip test if multipart uploads are not supported
+ Assume.assumeTrue("Skipping test as multipart uploads are not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ // Create a file with multipart upload but do not close the stream
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null, true);
+ stream.write(MULTIPART_FILE_BYTES);
+
+ // create and close another multipart file in parallel
+ createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true);
+
+ // Closing the first stream should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchOverwriteWithEmptyFile() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create a non-empty file
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // overwrite with zero-byte file (no write)
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null);
+
+ // close the stream, should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchOverwriteEmptyFileWithFile() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create an empty file (no write)
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null);
+ stream.close();
+
+ // overwrite with non-empty file, should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchOverwriteEmptyWithEmptyFile() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create an empty file (no write)
+ FSDataOutputStream stream1 = getStreamWithFlags(fs, testFile, true, null);
+ stream1.close();
+
+ // overwrite with another empty file, should throw RemoteFileChangedException
+ FSDataOutputStream stream2 = getStreamWithFlags(fs, testFile, true, null);
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream2::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfMatchOverwriteWithCorrectEtag() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path path = methodPath();
+ fs.mkdirs(path.getParent());
+
+ // Create a file
+ createFileWithFlags(fs, path, SMALL_FILE_BYTES, false, null);
+
+ // Retrieve the etag from the created file
+ String etag = ((S3AFileStatus) fs.getFileStatus(path)).getETag();
+ assertNotNull("ETag should not be null after file creation", etag);
Review Comment:
1. this code is duplicated across many tests. pull out into its own method (getEtag(path))
2. use AssertJ Assertions.assertThat().isNotNull();
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,564 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.FileNotFoundException;
+import java.nio.charset.StandardCharsets;
+
+import org.junit.Assume;
+import org.junit.Test;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileAlreadyExistsException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3AFileStatus;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.fs.s3a.Statistic;
+import org.apache.hadoop.fs.s3a.statistics.BlockOutputStreamStatistics;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE;
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_MULTIPART;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_OVERWRITE_SUPPORTED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_404_NOT_FOUND;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_412_PRECONDITION_FAILED;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1KB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private static final int UPDATED_MULTIPART_THRESHOLD = 100 * _1KB;
+
+ private static final byte[] SMALL_FILE_BYTES = dataset(TEST_FILE_LEN, 0, 255);
+ private static final byte[] MULTIPART_FILE_BYTES = dataset(UPDATED_MULTIPART_THRESHOLD * 5, 'a', 'z' - 'a');
+
+ private BlockOutputStreamStatistics statistics;
+
+ @Override
+ public Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+
+ S3ATestUtils.disableFilesystemCaching(conf);
+ removeBaseAndBucketOverrides(
+ conf,
+ MULTIPART_SIZE,
+ UPLOAD_PART_COUNT_LIMIT,
+ MIN_MULTIPART_THRESHOLD);
+ conf.setLong(UPLOAD_PART_COUNT_LIMIT, 2);
+ conf.setLong(MIN_MULTIPART_THRESHOLD, UPDATED_MULTIPART_THRESHOLD);
+ conf.setInt(MULTIPART_SIZE, UPDATED_MULTIPART_THRESHOLD);
+ return conf;
+ }
+
+ @Override
+ public void setup() throws Exception {
+ super.setup();
+ Configuration conf = getConfiguration();
+ skipIfNotEnabled(conf, FS_S3A_CREATE_OVERWRITE_SUPPORTED,
+ "Skipping IfNoneMatch tests");
+ }
+
+ private static void assertS3ExceptionStatusCode(int code, Exception ex) {
+ S3Exception s3Exception = (S3Exception) ex.getCause();
+
+ if (s3Exception.statusCode() != code) {
+ throw new AssertionError("Expected status code " + code + " from " + ex, ex);
+ }
+ }
+
+ protected String getBlockOutputBufferName() {
+ return FAST_UPLOAD_BUFFER_ARRAY;
+ }
+
+ private static void createFileWithFlags(
+ FileSystem fs,
+ Path path,
+ byte[] data,
+ boolean ifNoneMatchFlag,
+ String etag,
+ boolean forceMultipart) throws Exception {
+ FSDataOutputStream stream = getStreamWithFlags(fs, path, ifNoneMatchFlag, etag, forceMultipart);
+ if (data != null && data.length > 0) {
+ stream.write(data);
+ }
+ stream.close();
+ }
+
+ private static void createFileWithFlags(
+ FileSystem fs,
+ Path path,
+ byte[] data,
+ boolean ifNoneMatchFlag,
+ String etag) throws Exception {
+ createFileWithFlags(fs, path, data, ifNoneMatchFlag, etag, false);
+ }
+
+ private static FSDataOutputStream getStreamWithFlags(
+ FileSystem fs,
+ Path path,
+ boolean ifNoneMatchFlag,
+ String etag,
+ boolean forceMultipart) throws Exception {
+ FSDataOutputStreamBuilder builder = fs.createFile(path);
+ if (ifNoneMatchFlag) {
+ builder.must(FS_OPTION_CREATE_CONDITIONAL_OVERWRITE, "true");
+ }
+ if (etag != null) {
+ builder.must(FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG, etag);
+ }
+ if (forceMultipart) {
+ builder.opt(FS_S3A_CREATE_MULTIPART, "true");
+ }
+ return builder.create().build();
+ }
+
+ private static FSDataOutputStream getStreamWithFlags(
+ FileSystem fs,
+ Path path,
+ boolean ifNoneMatchFlag,
+ String etag) throws Exception {
+ return getStreamWithFlags(fs, path, ifNoneMatchFlag, etag, false);
+ }
+
+ private static String readFileContent(FileSystem fs, Path path) throws Throwable {
+ try (FSDataInputStream inputStream = fs.open(path)) {
+ return IOUtils.toString(inputStream, StandardCharsets.UTF_8);
+ }
+ }
+
+ private void updateStatistics(FSDataOutputStream stream) {
+ statistics = S3ATestUtils.getOutputStreamStatistics(stream);
+ }
+
+ @Test
+ public void testIfNoneMatchConflictOnOverwrite() throws Throwable {
+ describe("generate conflict on overwrites");
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create a file over an empty path: all good
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // attempted overwrite fails
+ RemoteFileChangedException firstException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, firstException);
+
+ // second attempt also fails
+ RemoteFileChangedException secondException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, secondException);
+
+ // Delete file and verify an overwrite works again
+ fs.delete(testFile, false);
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+ }
+
+ @Test
+ public void testIfNoneMatchConflictOnMultipartUpload() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip if multipart upload not supported
+ Assume.assumeTrue("Skipping as multipart upload not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true);
+
+ RemoteFileChangedException firstException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, firstException);
+
+ RemoteFileChangedException secondException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, secondException);
+ }
+
+ @Test
+ public void testIfNoneMatchMultipartUploadWithRaceCondition() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip test if multipart uploads are not supported
+ Assume.assumeTrue("Skipping test as multipart uploads are not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ // Create a file with multipart upload but do not close the stream
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null, true);
+ stream.write(MULTIPART_FILE_BYTES);
+
+ // create and close another small file in parallel
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // Closing the first stream should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchTwoConcurrentMultipartUploads() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip test if multipart uploads are not supported
+ Assume.assumeTrue("Skipping test as multipart uploads are not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ // Create a file with multipart upload but do not close the stream
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null, true);
+ stream.write(MULTIPART_FILE_BYTES);
+
+ // create and close another multipart file in parallel
+ createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true);
+
+ // Closing the first stream should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchOverwriteWithEmptyFile() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create a non-empty file
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // overwrite with zero-byte file (no write)
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null);
+
+ // close the stream, should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchOverwriteEmptyFileWithFile() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create an empty file (no write)
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null);
+ stream.close();
+
+ // overwrite with non-empty file, should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchOverwriteEmptyWithEmptyFile() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create an empty file (no write)
+ FSDataOutputStream stream1 = getStreamWithFlags(fs, testFile, true, null);
+ stream1.close();
+
+ // overwrite with another empty file, should throw RemoteFileChangedException
+ FSDataOutputStream stream2 = getStreamWithFlags(fs, testFile, true, null);
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream2::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfMatchOverwriteWithCorrectEtag() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path path = methodPath();
+ fs.mkdirs(path.getParent());
+
+ // Create a file
+ createFileWithFlags(fs, path, SMALL_FILE_BYTES, false, null);
+
+ // Retrieve the etag from the created file
+ String etag = ((S3AFileStatus) fs.getFileStatus(path)).getETag();
+ assertNotNull("ETag should not be null after file creation", etag);
+
+ String updatedFileContent = "Updated content";
+ byte[] updatedData = updatedFileContent.getBytes(StandardCharsets.UTF_8);
+
+ // overwrite file with etag
+ createFileWithFlags(fs, path, updatedData, false, etag);
+
+ // read file and verify overwritten content
+ String fileContent = readFileContent(fs, path);
+ assertEquals(
+ "File content should be correctly updated after overwriting with the correct ETag",
+ updatedFileContent,
+ fileContent
+ );
+ }
+
+ @Test
+ public void testIfMatchOverwriteWithOutdatedEtag() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path path = methodPath();
+ fs.mkdirs(path.getParent());
+
+ // Create a file
+ createFileWithFlags(fs, path, SMALL_FILE_BYTES, true, null);
+
+ // Retrieve the etag from the created file
+ String etag = ((S3AFileStatus) fs.getFileStatus(path)).getETag();
+ assertNotNull("ETag should not be null after file creation", etag);
+
+ // Overwrite the file. Will update the etag, making the previously fetched etag outdated.
+ createFileWithFlags(fs, path, SMALL_FILE_BYTES, false, null);
+
+ // overwrite file with outdated etag. Should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, path, SMALL_FILE_BYTES, false, etag));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfMatchOverwriteDeletedFileWithEtag() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path path = methodPath();
+ fs.mkdirs(path.getParent());
+
+ // Create a file
+ createFileWithFlags(fs, path, SMALL_FILE_BYTES, false, null);
+
+ // Retrieve the etag from the created file
+ String etag = ((S3AFileStatus) fs.getFileStatus(path)).getETag();
+ assertNotNull("ETag should not be null after file creation", etag);
+
+ // delete the file
+ fs.delete(path);
+
+ // overwrite file with etag. Should throw FileNotFoundException
+ FileNotFoundException exception = intercept(FileNotFoundException.class,
+ () -> createFileWithFlags(fs, path, SMALL_FILE_BYTES, false, etag));
+ assertS3ExceptionStatusCode(SC_404_NOT_FOUND, exception);
+ }
+
+ @Test
+ public void testIfMatchOverwriteFileWithEmptyEtag() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path path = methodPath();
+ fs.mkdirs(path.getParent());
+
+ // Create a file
+ createFileWithFlags(fs, path, SMALL_FILE_BYTES, false, null);
+
+ // overwrite file with empty etag. Should throw IllegalArgumentException
+ intercept(IllegalArgumentException.class,
+ () -> createFileWithFlags(fs, path, SMALL_FILE_BYTES, false, ""));
+ }
+
+ @Test
+ public void testIfMatchMultipartUploadWithRaceCondition() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip test if multipart uploads are not supported
+ Assume.assumeTrue("Skipping test as multipart uploads are not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ // Create a file with multipart upload but do not close the stream
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null, true);
+ stream.write(MULTIPART_FILE_BYTES);
+
+ // create and close another small file in parallel
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // Closing the first stream should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfMatchTwoMultipartUploadsRaceConditionOneClosesFirst() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip test if multipart uploads are not supported
+ Assume.assumeTrue("Skipping test as multipart uploads are not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ // Create a file and retrieve its etag
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, false, null);
+ String etag = ((S3AFileStatus) fs.getFileStatus(testFile)).getETag();
+ assertNotNull("ETag should not be null after file creation", etag);
+
+ // Start two multipart uploads with the same etag
+ FSDataOutputStream stream1 = getStreamWithFlags(fs, testFile, false, etag, true);
+ FSDataOutputStream stream2 = getStreamWithFlags(fs, testFile, false, etag, true);
+
+ // Write data to both streams
+ stream1.write(MULTIPART_FILE_BYTES);
+ stream2.write(MULTIPART_FILE_BYTES);
+
+ // Close the first stream successfully. Will update the etag
+ stream1.close();
+
+ // Close second stream, should fail due to etag mismatch
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream2::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfMatchCreateFileWithoutOverwrite() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // Create a file and retrieve the etag
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, false, null);
+ String etag = ((S3AFileStatus) fs.getFileStatus(testFile)).getETag();
+ assertNotNull("ETag should not be null after file creation", etag);
+
+ // Attempt to create a file at the same path without overwrite, using If-Match with the etag
+ FileAlreadyExistsException exception = intercept(FileAlreadyExistsException.class, () -> {
+ fs.createFile(testFile)
+ .overwrite(false)
+ .opt(FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG, etag)
+ .build();
+ });
+ }
+
+ @Test
+ public void testIfMatchCreateFileWithoutOverwriteWithPerformanceFlag() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ getConfiguration().set(FS_S3A_PERFORMANCE_FLAGS, "create");
+
+ // Create a file and retrieve the etag
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, false, null);
+ String etag = ((S3AFileStatus) fs.getFileStatus(testFile)).getETag();
+ assertNotNull("ETag should not be null after file creation", etag);
+
+ // Attempt to create a file at the same path without overwrite, using If-Match with the etag
+ FileAlreadyExistsException exception = intercept(FileAlreadyExistsException.class, () -> {
+ fs.createFile(testFile)
+ .overwrite(false)
+ .opt(FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG, etag)
+ .build();
+ });
+ }
+
+ @Test
+ public void testConditionalWriteStatisticsWithoutIfNoneMatch() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // write without an If-None-Match
+ // conditional_write, conditional_write_statistics should remain 0
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, false, null, false);
+ updateStatistics(stream);
+ stream.write(SMALL_FILE_BYTES);
+ stream.close();
+ long conditionalCreate = statistics.lookupCounterValue(Statistic.CONDITIONAL_CREATE.getSymbol());
+ long conditionalCreateFailed = statistics.lookupCounterValue(Statistic.CONDITIONAL_CREATE_FAILED.getSymbol());
+ assertEquals("Write without If-None-Match: conditional_create should be 0", 0L, conditionalCreate);
+ assertEquals("Write without If-None-Match: conditional_create_failed should be 0", 0L, conditionalCreateFailed);
+
+ // write with overwrite = true
+ // conditional_write, conditional_write_statistics should remain 0
+ try (FSDataOutputStream outputStream = fs.create(testFile, true)) {
+ outputStream.write(SMALL_FILE_BYTES);
+ updateStatistics(outputStream);
+ }
+ conditionalCreate = statistics.lookupCounterValue(Statistic.CONDITIONAL_CREATE.getSymbol());
+ conditionalCreateFailed = statistics.lookupCounterValue(Statistic.CONDITIONAL_CREATE_FAILED.getSymbol());
+ assertEquals("Write with overwrite=true: conditional_create should be 0", 0L, conditionalCreate);
+ assertEquals("Write with overwrite=true: conditional_create_failed should be 0", 0L, conditionalCreateFailed);
+
+ // write in path where file already exists with overwrite = false
+ // conditional_write, conditional_write_statistics should remain 0
+ try (FSDataOutputStream outputStream = fs.create(testFile, false)) {
+ outputStream.write(SMALL_FILE_BYTES);
+ updateStatistics(outputStream);
+ } catch (FileAlreadyExistsException e) {}
+ conditionalCreate = statistics.lookupCounterValue(Statistic.CONDITIONAL_CREATE.getSymbol());
+ conditionalCreateFailed = statistics.lookupCounterValue(Statistic.CONDITIONAL_CREATE_FAILED.getSymbol());
+ assertEquals("Write in path where file already exists with overwrite=false: conditional_create should be 0", 0L, conditionalCreate);
+ assertEquals("Write in path where file already exists with overwrite=false: conditional_create_failed should be 0", 0L, conditionalCreateFailed);
+
+ // delete the file
+ fs.delete(testFile, false);
+
+ // write in path where file doesn't exist with overwrite = false
+ // conditional_write, conditional_write_statistics should remain 0
+ try (FSDataOutputStream outputStream = fs.create(testFile, false)) {
+ outputStream.write(SMALL_FILE_BYTES);
+ updateStatistics(outputStream);
+ }
+ conditionalCreate = statistics.lookupCounterValue(Statistic.CONDITIONAL_CREATE.getSymbol());
+ conditionalCreateFailed = statistics.lookupCounterValue(Statistic.CONDITIONAL_CREATE_FAILED.getSymbol());
+ assertEquals("Write in path where file does not exist with overwrite=false: conditional_create should be 0", 0L, conditionalCreate);
+ assertEquals("Write in path where file does not exist with overwrite=false: conditional_create_failed should be 0", 0L, conditionalCreateFailed);
+ }
+
+ @Test
+ public void testConditionalWriteStatisticsWithIfNoneMatch() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null, false);
+ updateStatistics(stream);
+ stream.write(SMALL_FILE_BYTES);
+ stream.close();
+
+ long conditionalCreate = statistics.lookupCounterValue(Statistic.CONDITIONAL_CREATE.getSymbol());
Review Comment:
There's a class IOStatisticAssertions to help assert here...look at its usagesj. It's assertions result in AssertJ assertions, so you can go
```java
verifyStatisticCounterValue(statistics, CONDITIONAL_CREATE.getSymbol, 1)
```
same elsewhere
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Statistic.java:
##########
@@ -137,6 +137,12 @@ public enum Statistic {
TYPE_COUNTER),
IGNORED_ERRORS("ignored_errors", "Errors caught and ignored",
TYPE_COUNTER),
+ CONDITIONAL_CREATE(StoreStatisticNames.CONDITIONAL_CREATE,
+ "Count of successful conditional create operations.",
+ TYPE_COUNTER),
Review Comment:
put these in alphabetical order with all the other statistics
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,564 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.FileNotFoundException;
+import java.nio.charset.StandardCharsets;
+
+import org.junit.Assume;
Review Comment:
use assertJ assumeThat
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java:
##########
@@ -0,0 +1,564 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.impl;
+
+import java.io.FileNotFoundException;
+import java.nio.charset.StandardCharsets;
+
+import org.junit.Assume;
+import org.junit.Test;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FSDataOutputStreamBuilder;
+import org.apache.hadoop.fs.FileAlreadyExistsException;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.RemoteFileChangedException;
+import org.apache.hadoop.fs.s3a.S3AFileStatus;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.fs.s3a.Statistic;
+import org.apache.hadoop.fs.s3a.statistics.BlockOutputStreamStatistics;
+import org.apache.hadoop.fs.s3a.performance.AbstractS3ACostTest;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE;
+import static org.apache.hadoop.fs.Options.CreateFileOptionKeys.FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG;
+import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
+import static org.apache.hadoop.fs.s3a.Constants.FAST_UPLOAD_BUFFER_ARRAY;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_MULTIPART;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_CREATE_OVERWRITE_SUPPORTED;
+import static org.apache.hadoop.fs.s3a.Constants.FS_S3A_PERFORMANCE_FLAGS;
+import static org.apache.hadoop.fs.s3a.Constants.MIN_MULTIPART_THRESHOLD;
+import static org.apache.hadoop.fs.s3a.Constants.MULTIPART_SIZE;
+import static org.apache.hadoop.fs.s3a.Constants.STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.skipIfNotEnabled;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_404_NOT_FOUND;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.SC_412_PRECONDITION_FAILED;
+import static org.apache.hadoop.fs.s3a.impl.InternalConstants.UPLOAD_PART_COUNT_LIMIT;
+import static org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase._1KB;
+import static org.apache.hadoop.test.LambdaTestUtils.intercept;
+
+public class ITestS3APutIfMatch extends AbstractS3ACostTest {
+
+ private static final int UPDATED_MULTIPART_THRESHOLD = 100 * _1KB;
+
+ private static final byte[] SMALL_FILE_BYTES = dataset(TEST_FILE_LEN, 0, 255);
+ private static final byte[] MULTIPART_FILE_BYTES = dataset(UPDATED_MULTIPART_THRESHOLD * 5, 'a', 'z' - 'a');
+
+ private BlockOutputStreamStatistics statistics;
+
+ @Override
+ public Configuration createConfiguration() {
+ Configuration conf = super.createConfiguration();
+
+ S3ATestUtils.disableFilesystemCaching(conf);
+ removeBaseAndBucketOverrides(
+ conf,
+ MULTIPART_SIZE,
+ UPLOAD_PART_COUNT_LIMIT,
+ MIN_MULTIPART_THRESHOLD);
+ conf.setLong(UPLOAD_PART_COUNT_LIMIT, 2);
+ conf.setLong(MIN_MULTIPART_THRESHOLD, UPDATED_MULTIPART_THRESHOLD);
+ conf.setInt(MULTIPART_SIZE, UPDATED_MULTIPART_THRESHOLD);
+ return conf;
+ }
+
+ @Override
+ public void setup() throws Exception {
+ super.setup();
+ Configuration conf = getConfiguration();
+ skipIfNotEnabled(conf, FS_S3A_CREATE_OVERWRITE_SUPPORTED,
+ "Skipping IfNoneMatch tests");
+ }
+
+ private static void assertS3ExceptionStatusCode(int code, Exception ex) {
+ S3Exception s3Exception = (S3Exception) ex.getCause();
+
+ if (s3Exception.statusCode() != code) {
+ throw new AssertionError("Expected status code " + code + " from " + ex, ex);
+ }
+ }
+
+ protected String getBlockOutputBufferName() {
+ return FAST_UPLOAD_BUFFER_ARRAY;
+ }
+
+ private static void createFileWithFlags(
+ FileSystem fs,
+ Path path,
+ byte[] data,
+ boolean ifNoneMatchFlag,
+ String etag,
+ boolean forceMultipart) throws Exception {
+ FSDataOutputStream stream = getStreamWithFlags(fs, path, ifNoneMatchFlag, etag, forceMultipart);
+ if (data != null && data.length > 0) {
+ stream.write(data);
+ }
+ stream.close();
+ }
+
+ private static void createFileWithFlags(
+ FileSystem fs,
+ Path path,
+ byte[] data,
+ boolean ifNoneMatchFlag,
+ String etag) throws Exception {
+ createFileWithFlags(fs, path, data, ifNoneMatchFlag, etag, false);
+ }
+
+ private static FSDataOutputStream getStreamWithFlags(
+ FileSystem fs,
+ Path path,
+ boolean ifNoneMatchFlag,
+ String etag,
+ boolean forceMultipart) throws Exception {
+ FSDataOutputStreamBuilder builder = fs.createFile(path);
+ if (ifNoneMatchFlag) {
+ builder.must(FS_OPTION_CREATE_CONDITIONAL_OVERWRITE, "true");
+ }
+ if (etag != null) {
+ builder.must(FS_OPTION_CREATE_CONDITIONAL_OVERWRITE_ETAG, etag);
+ }
+ if (forceMultipart) {
+ builder.opt(FS_S3A_CREATE_MULTIPART, "true");
+ }
+ return builder.create().build();
+ }
+
+ private static FSDataOutputStream getStreamWithFlags(
+ FileSystem fs,
+ Path path,
+ boolean ifNoneMatchFlag,
+ String etag) throws Exception {
+ return getStreamWithFlags(fs, path, ifNoneMatchFlag, etag, false);
+ }
+
+ private static String readFileContent(FileSystem fs, Path path) throws Throwable {
+ try (FSDataInputStream inputStream = fs.open(path)) {
+ return IOUtils.toString(inputStream, StandardCharsets.UTF_8);
+ }
+ }
+
+ private void updateStatistics(FSDataOutputStream stream) {
+ statistics = S3ATestUtils.getOutputStreamStatistics(stream);
+ }
+
+ @Test
+ public void testIfNoneMatchConflictOnOverwrite() throws Throwable {
+ describe("generate conflict on overwrites");
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create a file over an empty path: all good
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // attempted overwrite fails
+ RemoteFileChangedException firstException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, firstException);
+
+ // second attempt also fails
+ RemoteFileChangedException secondException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, secondException);
+
+ // Delete file and verify an overwrite works again
+ fs.delete(testFile, false);
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+ }
+
+ @Test
+ public void testIfNoneMatchConflictOnMultipartUpload() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip if multipart upload not supported
+ Assume.assumeTrue("Skipping as multipart upload not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true);
+
+ RemoteFileChangedException firstException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, firstException);
+
+ RemoteFileChangedException secondException = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, secondException);
+ }
+
+ @Test
+ public void testIfNoneMatchMultipartUploadWithRaceCondition() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip test if multipart uploads are not supported
+ Assume.assumeTrue("Skipping test as multipart uploads are not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ // Create a file with multipart upload but do not close the stream
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null, true);
+ stream.write(MULTIPART_FILE_BYTES);
+
+ // create and close another small file in parallel
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // Closing the first stream should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchTwoConcurrentMultipartUploads() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+
+ // Skip test if multipart uploads are not supported
+ Assume.assumeTrue("Skipping test as multipart uploads are not supported",
+ fs.hasPathCapability(testFile, STORE_CAPABILITY_MULTIPART_UPLOAD_ENABLED));
+
+ // Create a file with multipart upload but do not close the stream
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null, true);
+ stream.write(MULTIPART_FILE_BYTES);
+
+ // create and close another multipart file in parallel
+ createFileWithFlags(fs, testFile, MULTIPART_FILE_BYTES, true, null, true);
+
+ // Closing the first stream should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchOverwriteWithEmptyFile() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create a non-empty file
+ createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null);
+
+ // overwrite with zero-byte file (no write)
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null);
+
+ // close the stream, should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchOverwriteEmptyFileWithFile() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create an empty file (no write)
+ FSDataOutputStream stream = getStreamWithFlags(fs, testFile, true, null);
+ stream.close();
+
+ // overwrite with non-empty file, should throw RemoteFileChangedException
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class,
+ () -> createFileWithFlags(fs, testFile, SMALL_FILE_BYTES, true, null));
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfNoneMatchOverwriteEmptyWithEmptyFile() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path testFile = methodPath();
+ fs.mkdirs(testFile.getParent());
+
+ // create an empty file (no write)
+ FSDataOutputStream stream1 = getStreamWithFlags(fs, testFile, true, null);
+ stream1.close();
+
+ // overwrite with another empty file, should throw RemoteFileChangedException
+ FSDataOutputStream stream2 = getStreamWithFlags(fs, testFile, true, null);
+ RemoteFileChangedException exception = intercept(RemoteFileChangedException.class, stream2::close);
+ assertS3ExceptionStatusCode(SC_412_PRECONDITION_FAILED, exception);
+ }
+
+ @Test
+ public void testIfMatchOverwriteWithCorrectEtag() throws Throwable {
+ FileSystem fs = getFileSystem();
+ Path path = methodPath();
+ fs.mkdirs(path.getParent());
+
+ // Create a file
+ createFileWithFlags(fs, path, SMALL_FILE_BYTES, false, null);
+
+ // Retrieve the etag from the created file
+ String etag = ((S3AFileStatus) fs.getFileStatus(path)).getETag();
+ assertNotNull("ETag should not be null after file creation", etag);
+
+ String updatedFileContent = "Updated content";
+ byte[] updatedData = updatedFileContent.getBytes(StandardCharsets.UTF_8);
+
+ // overwrite file with etag
+ createFileWithFlags(fs, path, updatedData, false, etag);
+
+ // read file and verify overwritten content
+ String fileContent = readFileContent(fs, path);
+ assertEquals(
Review Comment:
use assertJ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A createFile [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7329:
URL: https://github.com/apache/hadoop/pull/7329#issuecomment-2656823909
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ HADOOP-19256-s3-conditional-writes Compile Tests _ |
| +0 :ok: | mvndep | 6m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 19s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | compile | 19m 15s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 18m 2s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 4m 26s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | mvnsite | 2m 27s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | javadoc | 1m 58s | | HADOOP-19256-s3-conditional-writes passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | HADOOP-19256-s3-conditional-writes passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 49s | | HADOOP-19256-s3-conditional-writes passed |
| +1 :green_heart: | shadedclient | 37m 58s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 17m 55s | | the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 17m 55s | | the patch passed |
| +1 :green_heart: | compile | 17m 6s | | the patch passed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 17m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 4m 31s | [/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/artifact/out/results-checkstyle-root.txt) | root: The patch generated 142 new + 23 unchanged - 0 fixed = 165 total (was 23) |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| -1 :x: | javadoc | 1m 10s | [/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-common in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 50s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt) | hadoop-aws in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javadoc | 0m 46s | [/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/artifact/out/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06.txt) | hadoop-aws in the patch failed with JDK Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06. |
| +1 :green_heart: | spotbugs | 4m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 6s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 20m 35s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 55s | | hadoop-aws in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not generate ASF License warnings. |
| | | 250m 42s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9d450b01ee8e 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HADOOP-19256-s3-conditional-writes / 3e59b74729d5ccb601fe211c1189a1094aa28596 |
| Default Java | Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~us1-0ubuntu1~20.04-b06 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/testReport/ |
| Max. process+thread count | 1255 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7329/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org